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FOREWORD 


Historically,  Army  acquisition  research  has  had  difficulty  conducting  an  adequate  early 
assessment  of  the  human  dimension  in  system  performance.  Human  performance  research  is 
critical  to  Future  Combat  Systems  (FCS)  because  enhanced  battle  command  through  advanced 
command  and  control  (C^)  systems  is  at  the  heart  of  the  FCS  concept.  The  FCS  C^  program 
reflects  the  proactive  research  on  human  performance  needed  to  build  the  force  of  the  future. 
Attention  to  the  human  dimension  underscores  the  role  performed  by  the  U.S.  Army  Research 
Institute  for  the  Behavioral  and  Social  Sciences  (ARI)  as  the  Army’s  primary  research 
organi2ation  for  Persormel,  Training,  and  Leader  Development. 

This  report  describes  and  documents  ARI’s  work  and  products,  particularly  measurement 
methods  and  results,  as  a  key  member  of  the  Human  Performance  Team  for  the  FCS  C^  program. 
The  work  reported  here  focused  on  measuring  human  performance  to  understand  and  address 
task  and  training  requirements  for  command  groups  in  future  FCS  organizations.  This  report 
provides  sununary  results  and  conclusions  from  the  FCS  C^  experiments  that  attend  to  the  human 
dimension  of  battle  command  in  the  future  force. 

The  research  reported  here  reflects  ongoing  work  to  address  human  performance  issues 
by  ARI,  and  especially  the  Future  Battlefield  Conditions  (FBC)  Team  of  the  Armored  Forces 
Research  Unit  (AFRU).  This  report  supports  work  package  (211)  FUTURETRAIN:  Techniques 
and  Tools  for  Command,  Control,  Communications,  Computers,  Intelligence,  Surveillance,  and 
Reconnaissance  (C'^ISR)  Training  of  Future  Brigade  Combat  Team  Commanders  and  Staffs,  and 
supports  the  Science  and  Technology  Objective  (STO)  “Methods  and  Measures  of  Commander- 
Centric  Training.” 

Findings  from  this  effort  were  briefed  to  the  Deputy  Chief  of  Staff  for  Operations  and 
Training  (DCSOPS&T)  from  the  Training  and  Doctrine  Command  (TRADOC).  Methods  and 
findings  from  each  of  the  four  experiments  were  provided  to  the  Program  Manager  (PM)  for  FCS 
C9  as  part  of  ARJ’s  ongoing  support  to  FCS  and  Army  research  and  development  efforts.  Human 
performance  findings  by  ARI  helped  shape  the  C^  prototype  showcased  in  the  FCS  Capstone 
Demonstration  of  C^  systems  prior  to  the  FCS  Milestone  B  decision. 


— FRANKLIN  L.  MOSES 


Acting  Technical  Director 
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HUMAN  PERFORMANCE  ESSENTIAL  TO  BATTLE  COMMAND:  REPORT  ON  FOUR 
FUTURE  COMBAT  SYSTEMS  COMMAND  AND  CONTROL  (FCS  C^)  EXPERIMENTS 

EXECUTIVE  SUMMARY _ _ 


Research  Requirement: 

The  U.S.  Army’s  challenging  transformation  to  Future  Combat  Systems  (FCS)  entails  an 
unprecedented  amalgam  of  humans  and  machines,  a  truly  hybrid  fiiture  force.  A  pivotal  example 
of  the  FCS  transformation  challenge  is  the  requirement  that  a  relatively  small  command  group 
must  be  able  to  command  and  control  (C^)  an  interdependent  mix  of  maimed  and  autonomous 
systems.  An  ongoing  research  program  called  FCS  C^  exemplifies  the  Army’s  effort  to 
proactively  meet  this  requirement.  This  report  describes  and  documents  research  by  the  U.S. 
Army  Research  Institute  for  the  Behavioral  and  Social  Sciences  (ARI)  for  the  FCS  C^  program 
that  focuses  on  the  human  performance  essential  to  battle  command.  ^ 

Procedure: 

A  series  of  four  command-in-the-loop  exploratory  experiments  were  conducted  by  the 
FCS  C^  research  program  at  Fort  Monmouth,  NJ,  fi'om  October  2001  to  March  2003.  Primary 
participants  were  four  Active  Duty  Lieutenant  Colonels  who  formed  a  notional  FCS  command 
group  to  more  fully  explore  and  develop  new  command  and  control  paradigms.  Each  experiment 
lasted  two  weeks  with  the  first  three  days  dedicated  to  training  and  the  remaining  days  to 
experimental  exercises  referred  to  as  “runs.” 

Efforts  by  ARI  in  support  of  training  and  evaluation  resulted  in  the  respective  use  of 
deliberate  practice  and  run  complexity  levels.  Design  for  deliberate  practice  stressed  the 
repetition  of  similar  runs  with  feedback  to  ensure  results  were  based  on  proficient  performance. 
Run  complexity  was  varied  (Medium,  High,  and  Too  High)  to  assess  how  changes  in  operational 
conditions  might  impact  command  group  performance,  and  to  gauge  the  performance  limits  of 
the  proposed  Unit  Cell  organization. 

Measurement  methods  for  assessing  human  performance  were  developed  and  iteratively 
refined  by  ARI  across  the  four  experiments  to  better  understand  command  group  performance 
and  to  identify  training  requirements.  Subjective  measmes  obtained  participant  feedback  on 
multiple  research  issues  including:  training,  skill  proficiency,  workload,  performance  success, 
teamwork  skills,  decision  making,  function  and  task  allocations,  prototype  effectiveness,  and 
human-system  integration. 

Objective  measures  obtained  detailed  and  comprehensive  data  on  the  verbal  and  human- 
computer  interactions  performed  by  the  command  group  participants  during  run  planning  and 
execution  phases.  Analyses  related  the  relatively  micro  objective  behaviors  measured  to  more 
meaningful  functions  including  Plan,  See,  Move,  and  Strike. 


Vll 


Findings: 


Summary  description  and  documentation  on  the  human  performance  findings  obtained  by 
ARI  across  four  FCS  experiments  is  provided  in  this  report.  Overall,  the  body  of  subjective 
and  particularly  objective  results  obtained  on  human  performance  is  an  emerging  empirical 
database  on  command  group  task  and  training  requirements  in  small  FCS  units.  Measurement 
methods  and  the  evaluation  framework  developed  by  ARI  resulted  in  reliable  and  meaningftil 
data  on  humans  performing  command  and  control  in  a  notional  FCS  organization.  Such  human 
performance  data  is  needed  to  understand  and  improve  command  group  performance,  and 
particularly  to  address  training  issues  including:  task  analysis,  task  allocation,  workload, 
performance  assessment,  and  training  requirements. 

Results  and  discussion  are  followed  by  a  status  report  on  ARI’s  human  performance 
objectives  in  support  of  the  FCS  C^  research  program  to:  initiate  an  empirical  database  on  future 
command  group  performance  for  FCS;  measure  performance  to  iteratively  improve  the  FCS  C2 
research  environment;  develop  and  transfer  human  performance  measurement  methods  to  future 
efforts;  and  identify  training  requirements  and  develop  innovative  training  approaches  based  on 
empirical  measures  of  command  group  performance. 

Conclusions  stress  that  workload  and  training  requirements  for  future  command  groups 
represent  an  imposing  and  still  emerging  mix  of  technical  and  tactical  skills.  To  adequately 
achieve  these  skills  and  meet  the  FCS  requirements  for  simulation-based  and  embedded  training, 
more  exacting  measurement  methods  will  be  required  to  provide  the  feedback  needed  to  enable  ’ 
learning  and  improve  performance.  Conclusions  also  stress  the  potential  of  user-based 
involvement  and  proactive  research,  as  demonstrated  by  the  FCS  &  effort,  to  address  future 
battle  command  issues  particularly  workload  and  training  problems. 

Utilization  of  Findings: 

Methods  and  findings  on  human  performance  from  each  experiment  were  provided  to  the 
Program  Manager  (PM)  FCS  C^  as  part  of  ARI’s  ongoing  support  of  FCS  and  Army  research  and 
development  efforts.  Findings  by  ARI  were  briefed  to  the  Deputy  Chief  of  Staff  for  Operations 
and  Training  (DCSOPS&T)  from  the  Training  and  Doctrine  Command  (TRADOC).  The  FCS 
C  program  was  demonstrated  to  and  favorably  reviewed  by  the  U.S.  Army  Chief  of  Staff  and  the 
Secretary  of  the  Army  during  Experiment  4.  The  human  performance  findings  by  ARI  helped 
shape  the  C^  prototype  showcased  in  the  FCS  Capstone  Demonstration  of  C^  systems  prior  to  the 
FCS  Milestone  B  decision. 

The  measurement  methods  developed  by  ARI  to  measure,  analyze,  and  report  human 
performance  were  documented  to  facilitate  their  transfer  to  future  efforts,  particularly  research 
on  battle  command.  Training  issues  identified  will  guide  training  development  efforts  for  FCS 
including  ARI’s  Science  and  Technology  Objective  (STO)  titled  “Methods  and  Measures  of 
Commander-Centric  Training.” 
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HUMAN  PERFORMANCE  ESSENTIAL  TO  BATTLE  COMMAND:  REPORT  ON  FOUR 
FUTURE  COMBAT  SYSTEMS  COMMAND  AND  CONTROL  (FCS  C^)  EXPERIMENTS 

Introduction 

The  Army’s  ongoing  transformation  to  Future  Combat  Systems  (FCS')  poses  an 
unprecedented  alliance  of  humans  and  machines.  Examples  of  the  alliance  include  humans 
working  with  intelligent  agents  or  “hots”  for  information  processing  and  decision  aiding,  and 
with  robotic  entities  for  moving,  seeing,  and  shooting.  Creating  an  alliance  that  actually 
improves,  and  does  not  impede,  battle  command  is  a  human  systems  integration  challenge  for 
FCS. 


A  paradox  of  technology  is  that  advances  intended  to  ease  our  life  and  work  often  add 
complexity,  difticulty,  and  frustration  (Norman,  1988;  1997),  Humans  as  analog  systems  are 
compliant,  flexible,  and  tolerant;  in  contrast,  digital  systems  all  too  often  require  us  to  be  rigid, 
fixed,  and  intolerant.  This  paradox  is  of  particular  concern  to  the  design  and  development  of 
FCS  that  entails  an  extraordinary  mix  of  humans  and  machines,  a  truly  hybrid  future  force. 

Currently,  FCS  is  essentially  a  conceptual  design  featuring  an  interdependent  system-of- 
SN  stems  (Unit  of  Action  Maneuver  Battle  Laboratory,  2003).  Interdependence  is  reflected  in  the 
concept  of  a  network-centric  force  composed  of  modular  manned  and  progressively  autonomous 
platforms  with  netted  communication,  sensor,  and  fire  capabilities. 

I  low  to  best  allocate  human-machine  tasks  and  fimetions  for  command  groups  is  a 
growing  concern  for  FCS  force  transformation.  A  fundamental  lesson  from  modem  warfare  is 
that  the  in.sertion  of  technology  burdens  and  stresses  the  force  (Cordesman  &  Wagner,  1996). 
NotabK .  the  burden  on  humans  associated  with  advances  in  military  technology  is  attributed  less 
to  technology  per  se  than  to  inflated  expectations  about  technology. 

Cordesman  and  Wagner  (1996)  stressed  that  the  impact  of  heightened  expectations  on 
humans  is  pervasive  to  include:  technology  enables  faster  maneuver,  more  sorties,  extended 
fires,  just-in-time  logistics,  and  continuous  operations.  Moreover,  expectations  about  technology 
include  a  “more  with  fewer  personnel”  premise,  including  command  group  personnel. 

Although  automated  systems  are  begirming  to  relieve  commanders  and  staffs  of  some 
repetitive  and  computational  tasks,  the  human  tasks  that  remain  are  the  most  challenging  and 
critical  (Taylor,  Charlton  &  Canham,  1996,  p.301).  Many  command  and  control  tasks  are  too 
complicated  and  too  important  to  assign  to  machines.  For  battle  command,  for  example, 
advanced  technologies  should  help  commanders  visualize  the  operation,  describe  it  within  their 
intent,  and  direct  subordinates  toward  accomplishing  the  mission.  However,  the  “science”  of 
technology,  severely  lags  the  “art”  of  battle  command. 

A  pivotal  example  of  the  human-system  integration  challenge  in  FCS  is  the  requirement 
that  a  relatively  small  command  group  will  be  able  to  command  and  control  an  expansive  mix  of 
manned  and  autonomous  systems.  This  command  group  requirement  for  a  notional,  small 


'  A  list  of  acronyms  is  provided  in  Appendix  A. 
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combined  arms  echelon  in  future  FCS  organizations,  called  the  Unit  Cell  for  the  FCS  C~ 
program,  is  the  immediate  focus  of  the  research  discussed  here.  This  challenge  is  compounded 
by  the  command  group’s  extraordinary  reliance  on  technology,  and  by  the  expectation  that 
technology  will  enable  new  paradigms  in  command  and  control  (Unit  of  Action  Maneuver  Battle 
Laboratory,  2003).  In  sum,  for  FCS  the  integration  of  humans  and  automated  systems  is  integral 
to  both  re-conceptualizing  and  performing  command  and  control. 

The  Army  learns  by  doing.  The  research  reported  here  is  a  pioneering  effort  in  support  of 
the  Army’s  learning  requirement  for  new  approaches  to  battle  command.  Transforming  complex 
concepts  into  viable  solutions  requires  sustained  empirical  exploration,  assessment  and  feedback. 
Forging  a  forceful  human-machine  alliance  requires  shaping  technology  to  complement  human 
performance. 


Human  Performance  Perspective 

This  report  provides  summary  description  and  documentation  on  the  human  performance 
measurement  methods  developed  and  the  findings  obtained  in  the  Army’s  FCS  C^  program.  The 
FCS  C  program  reflects  the  Army’s  proactive  effort  to  understand  and  improve  performance  in 
emerging  FCS  organizations,  particularly  command  group  performance  in  small  units  with  a 
challenging  array  of  manned  and  unmanned  systems. 

This  report  examines  how  three  basic  principles  of  human  performance  apply  to  the  new 
challenges  raised  by  future  battle  command  requirements: 

■  Human  performance  is  essential  to  battle  command. 

■  Measurement  is  essential  to  understanding  human  performance. 

■  Training  is  essential  to  improving  human  performance. 

First  and  foremost,  human  performance  is  regarded  as  essential  to  battle  command.  The 
critical  role  of  the  human  dimension  in  military  operations  and  particularly  battle  command  is  a 
core  Army  value  and  a  doctrinal  precept  (Department  of  the  Army,  2001).  Attention  to  the 
human  dimension  also  underscores  the  role  performed  by  the  U.  S.  Army  Research  Institute  as 
the  Army’s  primary  research  organization  for  Personnel,  Training,  and  Leader  Development. 

The  unprecedented  alliance  of  humans  and  machines  by  FCS  reinforces  the  need  to  focus  on 
human  performance,  as  many  battle  command  tasks  remain  too  complex  and  too  important  to 
assign  to  machines  (Taylor,  Charlton  &  Canham,  1996,  p.  301). 

Second,  measurement  is  essential  to  understanding  human  performance.  Given  that 
battle  command  is  fundamentally  a  human  endeavor,  then  measurement  methods  are  needed  to 
understand  battle  command.  Human  performance  data  can  address  the  question  “How  well  did 
the  command  group  perform?”  versus  simply  asking,  “Who  won  the  battle?”  For  command- 
centered  research,  such  as  FCS  C^,  human  performance  data  are  more  relevant  and  valuable  than 
loss-exchange  ratios  and  other  battle  outcome  measures  for  assessing  the  new  approaches  to 
command  and  control  required  by  FCS. 
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A  more  fundamental  and  tractable  question  than  “How  well?”  is  “How?”  The  PCS  C 
efforts  by  ARI  focused  on  understanding  battle  command  by  objectively  measuring:  ""How  did 
the  command  group  perform  the  basic  functions  including  Plan,  See,  Move,  and  Strike  in  a 
futuristic  PCS  organization? 

Third,  training  is  essential  to  improving  human  performance.  A  critical  training  fallacy 
identified  by  Schneider  (1985)  is:  “Practice  makes  perfect.”  A  poorly  structured  regimen  of 
practice,  practice,  practice  often  results  in  little  or  no  improvement  in  learning  or  performance. 

A  more  structured  training  approach  improves  knowledge,  skills,  and  abilities  (Campbell, 
Quinkert,  &  Burnside,  2000).  Structured  training  also  makes  more  automatic  the  performance  of 
lower-level  skills  such  as  map  reading  and  terrain  visualization  (Pisk  &  Eboch,  1989)  needed  for 
higher-level  battle  command  skills  such  as  battlefield  visualization. 

A  form  of  structured  training  called  “deliberate  practice”  was  used  during  the  PCS  C^ 
experiments  to  stress  that  the  training  and  experimental  design  should  require  specific  objectives 
and  ensure  repeated  practice  opportunities  with  feedback.  Accordingly,  ARI’s  analysis  of  verbal 
and  human-computer  interaction  focused  on  latter,  more  deliberately  practiced,  exercises  or  runs 
from  each  experiment  to  better  ensure  findings  were  based  on  more  proficient  performance. 

The  need  to  measure  the  outcome  of  learning  to  determine  if,  or  ensure  that,  learning  has 
occurred  is  a  widely  accepted  practice  in  traditional  training  and  educational  settings.  Purposes 
for  outcome  measures  include  adjudicating  grades,  classes,  and  ranks  (pass/fail/percentile), 
determining  retention  rates,  detecting  remediation  needs,  and  prescribing  refi’esher  training. 

The  need  to  measure  the  process  of  learning  is  not  so  widely  appreciated  or  practiced. 
Process  measures  of  learning  are  needed  to  provide  the  feedback  instrumental  to  learning. 
Peedback  to  learners,  both  during  and  after  performing,  is  essential  to  learning  (Campbell, 
Quinkert,  &  Burnside,  2000;  Schneider,  1985). 

In  many  real  world  settings,  the  consequences  of  performance  often  provide  feedback 
automatically  and  clearly.  The  “measured”  comparison  between  actual  and  desired  performance 
may  be  so  obvious,  so  intuitive,  that  it  belies  any  measurement  requirement. 

However,  the  more  artificial  and  complex  the  performance  setting,  the  less  feedback  is  an 
automatic  affordance  and  the  more  difficult  its  interpretation.  Por  example,  in  the  simulation- 
based  runs  conducted  for  PCS  C^  with  interdependent  and  highly  automated  technologies, 
performance  feedback  was  too  often  missing  and  inadequate.  At  times,  automation  failed  to 
perform  as  expected.  At  other  times,  automation  performed  in  unexpected  and  detrimental  ways. 
The  causes  of  the  automation  problems  experienced  were  often  unclear.  Was  it  a  shortcoming  in 
the  technology,  in  the  training,  or  in  the  performers’  tactics,  techniques  and  procedures  (TTPs)? 

This  report  stresses  that  the  training  requirements  for  command  group  performance 
represent  an  imposing  and  still  emerging  mix  of  technical  and  tactical  skills.  To  adequately 
achieve  these  skills  and  the  new  PCS  training  requirements,  including  simulation-based  training 
and  embedded  training,  more  exacting  measurement  methods  will  be  required  to  provide  the 
feedback  needed  to  enable  learning  and  improve  performance. 
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Human  Performance  Research  Objectives 


The  major  research  objectives  that  directed  ARI’s  involvement  in  the  FCS  research 
program  are  bulleted  below  and  briefly  described: 

■  Initiate  an  empirical  database  on  future  command  group  performance  for  FCS. 

■  Measure  performance  to  iteratively  improve  the  FCS  C^  research  environment. 

■  Develop  and  transfer  human  performance  measurement  methods  to  future  efforts. 

■  Identify  training  requirements  and  develop  innovative  training  approaches  based  on 

empirical  measures  of  command  group  performance. 

A  primary  objective  of  ARI’s  efforts  to  measure  human  performance  was  to  initiate  the 
development  of  an  empirical  database  on  command  group  performance  for  future  FCS 
organizations.  The  command-in-the-loop  focus  of  FCS  C^  research  supported  the  requirement  to 
focus  on  the  measures  of  human  performance  needed  to  begin  to  understand  the  determinants 
and  consequences  of  command  group  behavior.  The  findings  from  this  exploratory  research 
program  were  expected  to  provide  empirical  groundwork,  if  not  benchmarks,  for  future  research 
on  battle  command  and  new  approaches  to  command  and  control. 

Another  more  immediate  objective  was  to  capture  and  document  the  performance  of  the 
command  group  during  each  experiment  in  order  to  iteratively  improve  the  overall  research 
en\’ironmcnt  for  latter  experiments.  Shortcomings  in  human-human  interaction  and  human- 
computer  interaction  were  routinely  identified  in  each  experiment.  Refinements  to  redress 
shortcomings  were  then  made  to  improve  system  design,  experimental  design,  and  training  in 
sub.scquent  experiments. 

A  more  general  objective  was  to  transfer  ARI’s  human  performance  measurement 
methods  to  future  efforts.  For  FCS  efforts,  the  goal  was  to  document  and  refine  the 
measurement  methods  in  a  marmer  that  other  researchers  could  apply  to  subsequent  FCS  C^ 
experiments.  To  facilitate  transfer,  ARI’s  documentation  included  methods  for  analyzing  and 
reporting  the  results  obtained  from  the  human  performance  measures.  For  example,  the  Interim 
Report  for  Experiment  3  provided  by  ARI  to  the  PM  FCS  C^  was  also  published  as  an  ARI 
Research  Report  (Lickteig,  Sanders,  Durlach,  &  Carnahan,  in  preparation)  to  ensure  more 
detailed  documentation  on  human  performance  measurement  methods  for  command  groups  is 
readily  available.  A  related  goal  was  to  provide  a  set  of  human  performance  measurement 
methods  that  would  transfer  to  a  broad  range  of  future  research  efforts  on  command  and  control, 
particularly  battle  command  and  command  group  research  issues. 

A  final  objective  was  to  identify  training  requirements  and  develop  innovative  training 
approaches  based  on  empirical  measures  of  command  group  performance.  This  goal  emphasizes 
the  crucial  role  of  performance  measurement  in  developing,  providing  and  evaluating  training  It 
also  directly  supports  the  ARI  STO  Objeetive  “Methods  and  Measures  of  Commander-Centric 
Training.”  The  evaluative  framework  for  measurement  developed  by  ARI  was  based  on  a 
functional  analysis  of  performance  that  related  relatively  micro  behaviors,  particularly  verbal  and 
human-computer  interaction,  to  relatively  macro  level  command  and  control  functions.  This  C^ 
function  approach  provided  a  meaningful  framework  to  compile  human  performance  data  that 
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could  be  used  to  address  training  related  issues  including:  task  analysis,  task  allocation, 
workload,  performance  assessment,  and  training  requirements. 

Overview  of  the  Evaluation  Framework 

An  overview  of  the  evaluation  framework  used  by  ARI  to  measure  and  assess  human 
performance  in  the  FCS  research  program  is  provided  here,  followed  by  a  more  detailed 
description  in  the  Method  section. 

The  functional  analysis  by  ARI  was  designed  to  identify  and  describe  the  command  and 
control  behaviors  of  the  command  group  for  a  small,  notional  FCS  unit  in  the  emerging 
Objective  Force  structure,  the  Unit  Cell.  Overall,  measurement  and  analysis  methods  for  human 
performance  developed  detailed  descriptions  of  critical  command  group  functions  and  supporting 
behaviors,  including  operational  definitions  and  behavioral  anchors,  that  applied  to  both  the 
planning  and  execution  phases  of  the  FCS  C^  experimental  runs. 

The  framework  for  measurement  was  based  on  a  functional  analysis  of  the  command 
group’s  performance  that  related  objective  behaviors  to  basic  C^  functions  adapted  from  the 
experimental  design  that  included  Plan,  See,  Move,  and  Strike.  The  evaluative  approach  for 
assessing  human  performance  was  based  on  the  following  measurement  methods: 

■  Subjective  Measures.  Responses  obtained  from  command  group  participants  in  after 
action  reviews  and  surveys  on  multiple  research  issues  including:  workload,  human 
performance,  system  performance  and  training. 

■  Verbal  Interactions.  Verbal  analysis  of  communications  included  transcription  from 
audio  recordings  of  all  spoken  exchanges  by  members  of  the  command  group  with  one 
another,  with  higher  headquarters,  and  with  subordinate  persoimel.  A  verbal 
communications  taxonomy  was  developed  as  a  structural  framework  for  the  verbal 
communications  analysis. 

■  Human-Computer  Interactions  (HCI).  The  HCI  analysis  of  participant  interactions  with 
the  prototype  included  comprehensive  analysis  of  all  interactions  across  the  command 
group  for  selected  runs.  An  HCI  taxonomy  was  developed  as  a  structural  framework  for 
HCI  analysis. 

■  Automated  Measures.  An  ARI  measurement  goal  was  to  promote  the  development  of 
automated  measures  of  C^  performance.  The  ARI  validated  the  available  subset  of 
automated  measures  developed  by  comparing  them  with  manual  measures  obtained  from 
HCI  analysis. 


The  FCS  Cf  Research  Environment 

The  FCS  C^  program  provides  a  working  example  of  the  Army’s  ongoing  effort  to 
explore  command  and  control  concepts  and  address  human  system  integration  issues  with  future 
command  group  performance.  From  October  2001  to  March  2003,  the  program  conducted  a 
series  of  four  command-in-the-loop  experiments  at  Fort  Morunouth,  NJ.  The  program  leads  are 
the  Defense  Advanced  Research  Projects  Agency  (DARPA)  and  the  U.S.  Army  Communications 
Electronics  and  Engineering  Command  (CECOM)  Research  and  Development  and  Engineering 


5 


Center  (RDEC).  As  a  partner  in  the  PCS  program,  ARI  focused  on  measuring  human 
performance  to  better  understand  and  address  performance  and  training  requirements  for  small 
command  groups  in  PCS  organizations. 

Purpose  of  PCS  Program 

The  stated  purpose  of  the  PCS  C2  program  is  to  explore  the  hypothesis  that  digitization 
of  current  battlefield  operating  systems  enables  a  new  approach  to  command  and  control: 

If  digitization  of  current  battlefield  operating  systems  can  substantially  enhance 
command  and  control  by  providing  better,  more  accurate,  and  timely  battlefield 
data  to  today’s  commander  and  staff  for  decision  making;  then  a  ‘new’  approach 
to  Battle  Command  and  Control,  implemented  in  the  form  of  synthesized/ 
analyzed  information  presented  to  the  future  Unit  Cell  Commander,  will  enable 
him  to  leverage  opportunities  by  focusing  on  fewer  unknowns,  clearly  visualizing 
current  and  future  end  states,  and  dictating  the  tempo  within  a  variety  of 
environments,  while  being  supported  by  a  significantly  reduced  staff  (Pronti, 

Molnar,  &  Wilson,  2002,  p.  ES-3). 

To  investigate  this  hypothesis,  the  PCS  C^  program  created  a  transformation  environment 
for  empirical  assessment  of  command  group  performance  at  the  small  unit  level,  the  notional 
Unit  Cell.  During  Experiments  1-4,  the  Unit  Cell  served  as  the  smallest  combined  arms  echelon 
within  PCS  organizations.  Pigure  1  depicts  the  manned  and  robotic  elements  of  the  Unit  Cell 
examined  in  Experiments  1-4,  including  the  surrogate  C^  vehicle  occupied  by  the  command 
group.  As  indicated  in  Pigure  1,  the  command  group  directly  controlled  16  air  and  ground 
vehicle  platforms,  13  of  which  were  unmanned,  and  a  set  of  unmanned  ground  sensors  (UGSs). 

Building  the  PCS  Transformation  Environment 

The  resources  and  products  of  three  interdependent  teams — Operational,  Technical,  and 
Human  Performance — were  required  to  create  the  PCS  C^  environment  for  transforming 
command  and  control.  The  Technical  Team  developed  the  Commander’s  Support  Environment 
(CSE),  the  hardware  and  software  system  located  in  the  surrogate  C^  vehicle.  The  CSE’s 
prototype  C  workstations  for  the  four  command  group  participants  allowed  them  to  command 
and  control  their  Unit  Cell  elements. 

The  Technical  Team  also  developed  supporting  technologies  such  as  the  Collaborative 
Server  so  the  command  group  could  share  information  via  a  common  operational  picture,  and  the 
Collective  Intelligence  module  to  ensure  the  Unit  Cell’s  elements  worked  together  in  a  network 
centric  environment.  Through  the  CSE’s  links  to  Distributed  Interactive  Simulation  (DIS),  the 
command  group  interacted  with  simulated  elements  of  the  Unit  Cell,  the  threat  force,  and  civilian 
entities. 
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Figure  1 .  Organization  of  the  Unit  Cell. 

The  Operational  Team  collaborated  on  prototype  design,  developed  mission 
requirements  and  scenarios  for  Unit  Cell  operations,  and  provided  participants  and  support 
personnel.  The  command  group  participants  were  four  active  duty  U.S.  Army  Lieutenant 
Colonels  in  the  duty  positions  of:  Commander,  Battle  Space  Manager,  Information  Manager, 
and  EITects  Manager.  The  Operational  Team  included  an  experienced  and  capable  group  of 
subject  matter  experts  (SMEs)  who  served  as  higher  Friendly  and  Enemy  commanders,  and  the 
Obser\  er/Controller  (0/C)  team  for  After  Action  Reviews  (AARs). 

The  Human  Performance  Team  devised  and  implemented  training  and  evaluation 
methods  compatible  with  an  incremental  series  of  experiments  designed  to  explore  and  document 
lessons  learned  for  Army  transformation  and  acquisition  objectives.  As  a  lead  member  of  this 
team,  ARl  efforts  focused  on  the  human-human  and  human-machine  interactions  required  by 
FCS  concepts  for  small  vmit  command  and  control.  A  core  tenet  stressed  by  ARI  was  that 
building  a  forceful  human-machine  alliance  required  shaping  technology  to  complement  hiunan 
performance. 

Formation  and  sustainment  of  a  proactive  research  environment  is  required  to  effectively 
integrate  human  and  system  performance.  Iterative  refinement  of  the  environment  and  research 
program  is  reflected  in  the  bulleted  schedule  for  Experiments  1-4  provided  below.  Experiment  1 
only  assessed  the  ability  of  the  Unit  Cell  to  move  its  elements  in  order  to  see  and  not  be  seen 
(i.e.,  See/Move).  By  Experiment  4,  Unit  Cell  missions  and  C^  prototype  capabilities  progressed 
to  Improved  See/Move/Strike  and  Sustain  operations. 
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■  Exp  1  —  Dec  2001  —  See/Move  (with  limited  Strike). 

■  Exp  2  —  May  2002  —  Improved  See/Move  and  Strike. 

■  Exp  3  —  Sep  2002  —  Improved  See/Move/Strike. 

■  Exp  4  —  Mar  2003  —  Improved  See/Move/Strike  and  Sustain. 

In  addition,  a  fifth  experiment  with  Army  Cadets  called  the  Summer  Experiment  was 
conducted  in  August  2002,  between  Experiment  2  and  3.  This  experiment  was  used  to  compare 
novice  versus  expert  command  groups  based  on  the  Cadet  performance  during  the  Summer 
Experiment  versus  the  Lieutenant  Colonels  performance  during  Experiments  2  and  3.  Results 
from  that  comparison  are  reported  in  an  ARI  Research  Report  by  Carnahan,  Lickteig,  Sanders, 
and  Durlach  (in  preparation). 


Method 

The  basic  experimental  schedule  for  PCS  C^  Experiments  1  -4  is  depicted  in  Table  I . 
Variations  in  schedule  were  relatively  moderate  for  an  exploratory  research  program  and 
primarily  due  to  technical  problems  during  Experiments  1-2.  This  summary  description  of 
research  methods  identifies  more  notable  variations  in  method  within  and  across  experiments. 
However,  the  program’s  deliberate  focus  on  new  approaches  to  battle  command  stressed 
exploration  and  iterative  refinement  of  system  and  human  parameters  across  experiments  to  meet 
emerging  operational  and  doctrinal  requirements  for  PCS. 

Table  1 


Basic  Experimental  Schedule  for  PCS  C^  Experiments  1-4 


Day 

Day 

Day 

Day 

Day 

Day 

Day 

Day 

Day 

Day 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

AM 

Training 

Run 

Run 

Run 

Run 

Run 

Run 

1 

3 

5 

7 

9 

11 

Pinal 

Run 

Run 

Run 

Run 

Run 

Run 

PM 

Training 

2 

4 

6 

8 

10 

12 

AAR 

AAR 

AAR 

AAR 

AAR 

AAR 

AAR 

Training  and  Experimental  Design 


Each  experiment  lasted  two  weeks  with  the  first  three  days  dedicated  to  training,  and  the 
remaining  days  to  experimental  exercises  referred  to  as  “runs.”  Training  on  days  1-2  of  each 
experiment  focused  on  developing  the  participants’  individual  skills  with  the  C^  prototype 
system.  This  training  was  intentionally  not  duty  position  specific,  but  rather  designed  to  provide 
cross-duty  skills  required  to  operate  the  C^  prototype  from  any  of  the  four  command  group  duty 
positions.  Training  on  the  third  day  typically  addressed  collective  command  group  skills  and 
took  place  in  the  surrogate  C^  vehicle  linked  by  virtual  simulation  to  friendly  and  enemy  entities 
to  support  run  rehearsal. 
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Experimental  runs  required  approximately  2-3  hours  to  complete  the  planning  and 
execution  phases,  resulting  in  two  runs  per  day  as  shown  in  Table  1.  At  the  end  of  each  run  day, 
the  0/C  team  led  an  AAR  of  that  day’s  runs  that  reviewed  operational,  technical,  and  human 
performance  issues.  More  complete  account  of  the  research  methods  is  provided  in  the  Interim 
Reports  on  each  experiment  provided  to  the  Program  Manager  (PM)  PCS  and  in  Lickteig, 
Sanders,  Durlach,  and  Carnahan,  in  preparation. 

Efforts  by  ARI  in  support  of  training  and  evaluation  resulted  in  the  respective  use  of 
deliberate  practice  and  run  complexity  levels.  Design  for  deliberate  practice  stressed  the 
repetition  of  similar  runs  with  feedback  to  ensure  results  were  based  on  proficient  performance. 
The  execution  phase  for  each  run  was  structured,  therefore,  and  limited  to  approximately  60-90 
minutes  to  help  maintain  focus  on  research  objectives.  Consequently,  the  operational  setting  and 
basic  mission  of  the  Unit  Cell  was  relatively  constant  across  experiments,  except  for  controlled 
variations  in  run  complexity. 

The  setting  was  simulated  desert  terrain  from  the  National  Training  Center  (NTC)  in 
which  the  Unit  Cell  conducted  deliberate  attack  missions  against  a  battalion  (minus/plus)  to  clear 
passage  lanes  for  a  follow-on  force.  The  performance  feedback  essential  to  deliberate  practice 
included  end-of-day  AARs.  Design  goals  were  to  help  participants  learn,  assess,  and  refine  the 
new  technical  skills  required  to  operate  their  C^  prototypes  and  the  new  tactical  skills  required  to 
exploit  the  Unit  Cell’s  progressively  automated  assets. 

Experimental  design  varied  run  complexity  levels  as  a  function  of  METT-TC  (Mission, 
Enemy,  Terrain,  Troops,  Time,  and  Civilians)  for  Experiments  1-3.  Three  complexity  levels 
(Medium,  High,  and  Too  High)  were  varied  by  increasing  enemy  force  activity  and  size, 
eliminating  a  key  friendly  asset,  and  inserting  civilians  on  the  battlefield. 

Run  complexity  was  introduced  to  identify  how  changes  in  operational  conditions  impact 
PCS  units,  particularly  command  group  performance,  and  to  gauge  the  performance  limits  of  the 
proposed  Unit  Cell  organization.  It  was  hypothesized  that  run  complexity  would  change 
workload,  performance,  and  human-system  integration  requirements.  To  establish  baseline 
performance  indicators  for  future  efforts,  however,  all  runs  for  Experiment  4  were  conducted  at 
the  “High”  level  of  complexity. 


Participants 

Four  Active  Duty  Lieutenant  Colonels  were  selected  as  command  group  participants  to 
more  fully  explore  and  develop  new  command  and  control  paradigms.  Alternate  participants 
were  used  occasionally,  however,  due  to  schedule  conflicts.  Notably,  the  Information  Manager 
for  the  latter  half  of  the  runs  in  Experiment  3  and  all  Experiment  4  runs  was  a  Major  with  less 
operational  and  command  experience.  However,  the  Major  had  extensive  experience  with  the 
PCS  C^  prototype  as  he  was  assigned  to  Fort  Monmouth  and  the  PCS  C^  research  program. 

Across  experiments,  the  participants  became  relatively  expert  members  in  an  PCS 
command  group.  This  assessment  of  expertise  is  based,  in  part,  on  an  assumption  that  command 
group  expertise  entails  an  incremental  progression  of  skill  development  from  tactical  skills,  to 
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technical  skills,  and  then  to  the  integration  of  tactical  and  technical  skills.  Clearly,  more  basic  or 
generic  technical  skills  such  as  familiarity  with  commercial  software  applications  may  support 
and  precede  tactical  skill  development.  However,  more  applied  technical  skills  on  how  to  use  a 
C  system  to  effectively  command  and  control  manned  and  unmanned  systems  are  only  tools  to 
support  the  “art”  of  battle  command.  Technical  tools  cannot  be  effectively  applied  without 
underlying  tactical  skills. 


The  tactical  expertise  of  the  primary  participants  included  exceptional  educational, 
professional  development,  and  operational  experience.  For  example,  their  selection  as 
participants  was  based  in  part  on  graduation  from  the  School  of  Advanced  Military  Studies 
(SAMS).  Participants  tactical  expertise  provided  a  solid  base  for  developing  and  applying  the 
technical  skills  required  to  command  and  control  the  assets  of  the  Unit  Cell  by  means  of  the  FCS 
C  prototype  interface. 

Technical  skills  were  developed  in  individual  and  collective  training  sessions  prior  to 
each  experiment’s  runs,  as  described  below  in  the  Training  section.  Participant  feedback  to 
system  developers  during  each  experiment  was  routinely  used  to  develop  and  revise  the 
system  and  interface,  including  the  addition  of  new  features  and  tools.  Many  of  the  technical 
skill  requirements,  therefore,  were  precisely  tailored  to  participants’  expectations  about  how 
features  and  tools  should  be  designed  and  applied.  Skill  integration  opporhinities  included  over 
40  +  experimental  runs  across  Experiments  1-4.  Notably,  these  runs  were  followed  by  After 
Action  Reviews  that  provided  tactical  and  technical  feedback  to  the  participants. 

The  integral  role  performed  by  key  support  personnel  suggests  they  be  regarded  as 
background  participants  essential  to  the  effort.  Three  (3)  support  personnel  roles  performed  by 
highly  experienced  SMEs  were:  higher  Friendly  commander.  Enemy  commander,  and  senior 
member  of  the  0/C  team.  The  expertise  and  vision  of  these  background  participants  in 
preparing,  supporting,  and  particularly  in  leading  the  reviews  of  experimental  runs  in  AARs  was 
invaluable  in  helping  the  participants  develop,  assess,  and  refine  new  approaches  to  command 
and  control. 


Prototype  Functions  and  Features 

Figure  2  depicts  a  sample  view  of  a  prototype  interface  used  in  FCS  C^  experiments. 
Dual  C  interfaces  were  located  at  each  of  the  four  participant  workstations  in  the  surrogate  C^ 
vehicle.  These  interfaces  provided  a  near  real-time  common  operational  picture  of  the  Unit 
Cell  s  battlefield  situation,  such  as  the  Deliberate  Attack  mission  against  an  enemy  battalion 
shovm  in  Figure  2.  The  C  interfaces  were  the  primary  means  by  which  the  command  group 
performed  basic  C  functions  such  as  Plan,  See,  Move,  and  Strike.  Participants  relied  on  their  C^ 
systems  to  command  and  control  the  Unit  Cell  assets  depicted  in  Figure  1,  particularly  to  task 
ground  and  air  robotic  platforms  and  receive  information  from  a  mix  of  ground  and  air  sensors. 

As  portrayed  in  Figure  2,  the  C"  display  was  based  on  a  relatively  standard  interface 
design  of  windows,  icons,  menus,  and  pull  downs  (WIMP).  The  set  of  windows  opened  in  the 
Figure  2  indicate  some  of  the  C^  sptem’s  functionality.  More  precisely,  the  interface  shown  in 
Figure  2  reflects  the  level  of  functionality  and  a  sample  of  the  new  features  available  to  the 
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Tactical  Map  with  Unit  Cell  in  Deliberate  Attack  Against  Red  Battalion  (Open  Terrain) 

Figure  2.  Sample  view  of  a  prototype  interface. 

command  group  by  Experiment  3.  Overall,  these  and  numerous  other  tools  and  features  could  be 
opened,  configured,  and  used  by  each  participant  during  the  course  of  a  run  or  mission.  The  four 
windowed  features  opened  in  Figure  2  are  briefly  described  below: 

■  Battlefield  Assistant  provided  information  regarding  various  alerts  that  participants  could 
tailor  to  their  information  requirements  and  activate.  For  example,  alerts  could  signal  the 
participants  when  a  previously  identified  and  tracked  fiiendly,  enemy,  or  neutral  vehicle 
had  crossed  a  phase  line  or  entered  a  named  area  of  interest  (NAJ). 

■  Image  Viewer  displayed  target  images  for  human  target  recognition  (HTR)  and  battle 
damage  assessment  (BDA).  Image  Viewer  allowed  participants  to  input  or  revise  the 
information  associated  with  icons  on  the  map  including  affiliation  (e.g.,  fiiendly,  enemy, 
neutral),  type  (e.g.,  tank,  artillery),  and  status  (e.g.,  suspected  target,  dead  target). 

■  Target  Catalog  allowed  participants  to  input  or  revise  information  associated  with  enemy 
targets  including  what  fiiendly  sensor(s)  had  identified  the  target,  when  the  target  was 
identified,  and  the  location  and  route  of  the  enemy  target. 

■  Resource  Availability  was  used  to  access  and  revise  information  on  fiiendly  Unit  Cell 
assets  including  operational  status,  available  fuel,  and  current  speed  and  location. 

Instrumentation 

An  adequate  assessment  of  human  performance  requires  a  balance  between  subjective 
measures  about  performance  and  objective  measures  of  performance.  The  ARI’s  measurement 


11 


methods  used  to  assess  human  performance  included  a  battery  of  subjective  measures  and 
detailed,  comprehensive  objective  measures  of  verbal  and  human-computer  interaction. 

Subjective  Measures 

The  set  of  subjective  measures  used  to  assess  human  performance  expanded  from  two  to 
ten  questionnaires  across  experiments.  Overall,  these  measures  were  designed  and  developed  to 
obtain  participant  feedback  on  multiple  research  issues  including:  training,  skill  proficiency, 
workload,  performance  success,  teamwork  skills,  decision  making,  function  and  task  allocations, 
prototype  effectiveness,  and  human-system  integration. 

A  brief  summary  of  the  basic  administration  schedule  is  provided  before  describing  some 
of  the  subjective  measures  used.  Most  questionnaires  were  administered  only  once  during  each 
experiment.  However,  several  questionnaires  such  as  skill  proficiency  and  training  adequacy 
were  administered  more  often  to  detect  changes  across  an  experiment.  For  example,  during 
Experiment  4  the  training  questionnaire  was  administered  before  and  after  the  participants 
completed  the  set  of  experimental  runs.  Only  the  questionnaires  on  workload  and  performance 
success  were  routinely  administered  after  each  run.  By  Experiment  4,  subjective  measures  were 
administered  electronically  rather  than  in  paper-and-pencil  format. 

Descriptions  of  selected  subjective  measures  are  provided  below  to  help  clarify  their 
corresponding  findings  reported  in  the  Results  section.  An  overview  list  and  description  of  all 
the  questionnaires  used  by  ARI  during  the  experiments  is  provided  in  Table  2.  A  complete  set  of 
all  subjective  measures  and  sample  results  from  each  measure  obtained  during  Experiment  3  is 
provided  in  an  ARI  Research  Report  (Lickteig,  Sanders,  Durlach,  &  Carnahan,  in  preparation). 

Workload  and  performance  success.  After  an  In-Place  AAR  as  depicted  in  Figure  3, 
participants  exited  the  Vehicle  and  completed  a  brief  survey  on  Workload  and  Performance 
Success.  Participants  rated  their  perceived  workload  across  five  dimensions:  Mental,  Physical, 
Temporal,  Effort,  and  Frustration  (1  =  Low  to  100  =  High).  The  workload  questions  and 
dimensions  were  adapted  from  the  relatively  standard  Task  Load  Index  (TLX)  from  the  National 
Aeronautics  and  Space  Administration  (NASA-Ames  Research  Center,  1986).  Performance 
success  by  duty  position  was  rated  on  the  TLX  questionnaire  in  an  item  worded  “How  successful 
were  you  in  accomplishing  what  you  were  asked  to  do?  (1  =  Failure  to  100  =  Perfect). 

New  Cf  prototype  features  effectiveness.  This  questionnaire  was  developed  to  more 
precisely  assess  the  value  of  the  new  C^  protot)q5e  features  added  for  Experiment  3.  It  asked  the 
participants  to  rate  the  effectiveness  of  13  new  prototype  features  on  a  5-point  scale  (1  =  Very 
Ineffective  to  5  =  Very  Effective).  The  questionnaire  was  administered  after  earlier  and  later 
runs  to  determine  if  perceived  effectiveness  of  the  new  features  changed  v^ath  experience.  This 
questiormaire  was  adapted  and  administered  by  a  supporting  contractor  during  Experiment  4. 
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Table  2 


Description  of  Subjective  Measures  by  Item  Content  and  Response  Format 


Subjective  Measure 

Item  Content 

Response  Format 

♦Workload  Task  Load  Index 
(TLX)  (NASA-Ames,  1986) 

Mental,  physical,  temporal,  effort,  and 
finstration  scales. 

(I  =  Low  to  100  =  High) 

♦Performance  Success 

How  successful  were  you...? 

(1  =  Low  to  100  =  High) 

*C^  Prototype  New  Features 
Effectiveness 

How  effective  were  the  new  C^ 
prototype  features  listed...? 

(1  =  Very  Ineffective  to 

5  =  Very  Effective) 

Prototype  New  Features 
Workload 

How  did  the  new  C^  prototype  features 
impact  workload? 

(1  =  Increased  Greatly  to 

5  =  Decreased  Greatly) 

Prototype  Support  of  Human- 
Computer  Interactions 

How  effective  was  the  C^  prototype  in 
support  of  the  C^  interactions  listed  ...? 

(1  =  Very  Ineffective  to 

5  =  Very  Effective) 

Prototype  Effectiveness  for 
Functions  and  Mission,  Enemy, 
Terrain,  Troops,  Time  and 

Civilians  (METT-TC)  Factors 

How  effective  was  the  C^  prototype  for 
C‘  Functions  and  METT-TC  Factors 
listed  below...? 

(1  =  Very  Ineffective  to 

5  =  Very  Effective) 

C^  Teamwork  Skills 

Provide  examples  of  effective  and 
ineffective  teamwork  skills  for  the  C^ 
tasks  listed  below  ... 

Open-ended  comments 

C^  Decision  making 

Describe  important  decisions  you  made 
how  C^  features  supported  each 
decision... 

Open-ended  comments 

Skills  Proficiency 

Rate  your  individual  and  the  group’s 
collective  proficiency. 

(1  =  Extremely  Low  to 

9  =  Extremely  High) 

♦Training  Adequacy 

How  adequate  was  training  content  and 
time  for  individual  and  collective  skills? 

Open-ended  comments 

Human  Systems  Integration 
(Durbin,  2002) 

HCI  workload  and  difficulty. 

Usability  of  the  prototype. 

Rating/checklist  formats 
Open-ended  for  problems  and 
recommendations. 

♦In-Place  AAR  (After  Action 
Review)  Interview 

What  went  right  and  what  went  wrong? 

Open-ended  comments 

Note.  *  Indicates  sample  measures  described  and  sample  findings  provided  in  this  report.  All  subjective  measures 
and  sample  results  from  Experiment  3  are  provided  in  Lickteig,  Sanders,  Durlach,  &  Carnahan,  in  preparation. 

Training.  A  training  adequacy  questionnaire  was  routinely  administered  after  the 
participants  had  completed  all  individual  and  collective  training  and  before  their  first 
experimental  run.  Hence,  it  was  called  a  Pre-Run  survey.  For  Experiment  4,  the  same  training 
adequacy  questionnaire  was  also  administered  after  the  participants  had  completed  all  of  the 
experimental  runs,  called  a  Post-Run  survey.  Four  open-ended  items  asked  participants  to  assess 
the  adequacy  of  the  individual  and  collective  training  in  terms  of  content  and  time  allotted. 

Objective  Measures 

Instrumentation  for  objective  measures  of  human  performance  primarily  included  video/ 
audio  recordings  of  all  experimental  runs  from  multiple  perspectives.  Recordings  included  a 
separate  wide-angle  view  of  each  participant  taken  from  across  the  interior  of  the  vehicle. 

The  wide-angle  perspectives  were  combined  into  a  quadraplex  recording  of  the  participants  that 
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greatly  assisted  verbal  analysts  in  transcribing  and  coding  verbalizations  into  pre-defined 
categories,  such  as  the  source  and  function  of  each  communication. 


Figure  3.  In-Place  AAR  conducted  in  surrogate  vehicle. 

In  addition,  eight  separate  recordings  provided  a  dedicated  view  for  each  of  the  two 
interfaces  at  each  of  the  four  participant  workstations  in  the  vehicle.  These  recordings  of  each 

C  interface  were  reviewed  extensively  by  the  human-computer  interaction  analysts  to  code  and 
tabulate  all  participant  interactions  with  their  prototypes.  Unfortunately,  low-resolution  video 
prevented  HCI  analysis  for  Experiment  1 .  Therefore,  instrumentation  upgrades  were  made  to 
digital  format  for  the  eight  interface  recordings  for  Experiments  2-4. 

During  each  experimental  run,  these  and  additional  recordings  were  streamed  to  multiple 
locations  throughout  the  FCS  testbed  to  support  observation  by  members  of  the  Operational, 
Techmcal,  and  Human  Performance  teams,  as  well  as  to  a  Visitors  Observation  room. 

By  Experiment  4,  refinement  of  objective  measurement  methods  added  a  dedicated  suite 
of  observer  workstations  that  allowed  four  observer  teams  to  more  closely  monitor  the  four 
participant  workstations.  The  four  displays  available  at  each  observer  workstation  provided  on- 
call  access  to  any  of  five  different  perspectives:  the  two  participant  interfaces  for  the  duty 
position  they  were  assigned  to  observe,  the  Heads-Up  Display  near  the  front  of  the  vehicle, 
the  “ground  truth”  battlefield  situation  from  the  One  Semi-Automated  Forces  Testbed  Baseline 
(OneSAF  0TB),  and  the  quadraplex  overview  of  the  four  command  group  participants.  Data 
lo8§crs  also  captured  the  performance  of  simulated  friendly  and  threat  entities  during  exercises 
and  a  small  subset  of  automated  measures  on  participant  interactions  with  their  prototypes. 


This  overall  suite  of  instrumentation  for  objective  measures  resulted  in  the  two  types  of 
objective  human  performance  data,  verbal  and  human-computer  interactions,  described  in  the 
subsequent  section  “Evaluation  Framework.”  For  Experiments  3  and  4,  ARI’s  analysis  of  these 
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objective  measures  during  the  execution  phase  was  extended  to  include  the  planning  phase  for 
selected  experimental  runs. 

Automated  Measures 

Automated  measures  of  C^performance  are  required  to  support  training,  evaluation,  and 
system  design  (Unit  of  Action  Maneuver  Battle  Laboratory,  2003).  Notably,  this  new  PCS 
requirement  for  automated  measures,  particularly  measures  of  human-computer  interaction, 
applies  to  PCS  prototype  and  fielded  systems. 

2 

An  ARI  measurement  goal  was  to  promote  the  development  of  automated  measures  of  C 
performance.  Only  minimal  progress  toward  that  goal  was  made  during  the  PCS  experiments. 

However,  the  approach  taken  and  lessons  learned  are  documented  here  to  underscore  the  work 
required  and  payoff  available  by  automating  many  of  the  human-machine  performance  measures 
integral  to  future  command  and  control. 

Automated  measures  of  human-computer  interaction  can  provide  efficient  and  effective 
measures  of  command  and  control  performance.  The  efficiency  of  automated  measures  equates 
to  quick  and  inexpensive.  It  includes  the  ability  to  adjust  the  range  and  selection  of  data  to 
include  the  performance  of  any  or  all  users  at  any  or  all  times  during  an  operational  exercise. 
The  effectiveness  of  automated  measures  equates  to  increased  scope  and  precision  in  Ae 
collection  of  performance  data.  It  includes  more  meaningful  measures  by  automatically 
correlating  performance  with  the  battlefield  situation  in  which  it  occurred. 

In  contrast,  manual  data  capture  and  reduction  of  command  and  control  performance  can 
only  examine  a  fraction  of  the  data  available  from  any  training,  testing  or  evaluation  effort.  The 
burden  in  labor  and  time  to  provide  objective  and  direct  measures  of  command  and  control 
performance  is  practically  imbearable  (Brown,  Nordyke,  Gerlock,  Begley,  &  Meliza,  1998; 
Lickteig  &  Quinkert,  2001).  Manual  measures  are  unresponsive  to  pressing  timelines,  including 
training  feedback  and  performance  support  requirements.  The  manual  measurement  methods 
reported  here  to  obtain  objective  data  on  performance  attest  to  that  burden,  and  limited  the 
analysis  of  all  verbal  and  human-computer  interactions  by  the  command  group  to  selected  runs. 

After  Experiment  2  and  ARI’s  first  analysis  of  HCI  performance,  ARI  requested  that  a 
set  of  key  automated  measures  be  developed  to  support  human  performance  assessment.  Table  3 
lists  and  describes  the  23  automated  measures  requested  to  capture  command  group  interactions 
with  their  prototypes.  Validation  methods  compared  the  automated  data  from  Experiment  3 
with  ARI’s  manually  derived  data  on  corresponding  HCI  measures. 

Evaluation  Framework 

The  human  performance  issues  for  future  battle  command  were  based  on  a  functional 
analysis  of  the  Unit  Cell’s  command  group  performance.  The  set  of  C^  functions  adapted  fi-om 
the  experimental  design  for  analyzing  verbal  and  human-computer  interactions  is  bulleted  below: 
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■  Plan:  Develop,  assess,  and  modify  plans  including  tasking  for  unmanned  air/ground 
assets  in  response  to  changing  events. 

■  See:  Control/interpret  input  from  set  of  manned/unmanned  networked  sensors  to 
maintain  accurate  battlefield  “picture.” 

■  Move:  Control  movement/activity  manned/immanned  assets. 

■  Strike:  Control  mamial/automated  networked  fires. 

■  BDA:  Control/interpret  input  from  set  of  manned/unmaimed  networked  sensors  for  battle 
damage  assessment  (BDA). 

A  method  inconsistency  noted  is  that  BDA  was  categorized  as  separate  function  for 
verbal  analysis,  but  as  a  sub-function  under  the  See  function  for  analysis  of  human-computer 
interactions. 

Table  3 

Automated  Measures  Requested  by  ARI  for  Experiment  3 

Measurement  Category 
Automated  Measure 

See  ^  ~~  “ 

Number  of  pictures/images  available. 

Number  of  pictures/images  with  actual  enemy  image  of  those  available. 

Number  of  pictures/images  opened. 

Amount  of  time  manipulating  pictures  (zoom,  contrast,  pan)  to  improve  image. 

Number  of  times  same  picture  opened  by  same  individual. 

Number  of  times  same  picture  opened  by  different  individuals. 

Alerts 

Number  and  type  of  alerts  set  by  duty  position. 

Number  and  type  of  alerts  triggered/activated  by  duty  position. 

Time  to  respond  to  alerts  by  turning  it  off. 

Number  of  times  robotic  vehicles  auto  halt  (red  line  under  the  vehicle). 

Time  to  respond  to  auto  halt  to  re-tasking  the  entity. 

Number  of  fratricide  warnings  (identify  shooter/target  pair). 

Move  Assets 

Number  of  times  and  task  duration  for  Create  a  Route  (ground  platform). 

Number  of  times  and  task  duration  for  Create  a  Route  (air  platform). 

Strike 

Number  of  times  weapon  fired  and  task  duration  (Netfire,  Line  of  Sight  (LOS),  Infantry  Fighting  Vehicle  [IFV]). 
Number  of  times  “Reassign”  menu  option  selected  for  Loitering  Attack  Missile  (LAM). 

Number  of  times  target  Type  changed  by  selecting  “Apply”  button  in  Recon  Window. 

Number  of  times  Target  Status  modified  by  selecting  “Suspected,”  “Targeted,”  “Dead,”  etc. 

Number  of  times  sensors  tasked  to  recon  targets  by  selecting  “OK”  from  Recon  Target  menu. 

Assess  Icon  and  Map  Information 

Number  of  times  cursor  moved  over  a  platform  to  bring  up  information  window. 

Number  of  times  “Templated”  targets  toggled  off  in  status  toolbar. 

Number  of  times  Zoom  Map  options  selected  (box  and/or  arrows  menu  option). 

Number  of  times  scroll  arrows  used  for  map  manipulation. 
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Verbal  Analysis 

Verbal  interactions  from  selected  runs  were  analyzed  based  on  transcribed  voice 
communications  among  the  command  group  participants,  and  the  participants’  communications 
with  the  higher  and  lower  elements  role  played  by  supporting  personnel.  The  verbal  transcripts 
were  first  “chunked”  into  more  meaningful  blocks  of  communication  for  subsequent  coding. 
Chunking  required  researchers  to  evaluate  the  communications  and  then  group  a  cluster  of  dialog 
together  that  appeared  to  be  imitary  and  consistent.  The  goal  of  chunking  was  to  create  coherent 
blocks  of  dialog  that  were  specific  enough  in  they  did  not  fall  imder  multiple  rating  categories. 

Chunking  and  classifying  communications  into  meaningful  categories  often  requires  a 
consideration  of  context  and  a  degree  of  interpretation  that  broadens  disagreement  between  raters 
independently  assigning  codes  to  each  verbal  chunk.  If  raters  fail  to  agree  in  their  coding,  then 
one  must  question  the  reliability  of  the  method  and  validity  of  the  results.  One  way  to  increase 
inter-rater  agreement  is  to  identify  smaller  chunks  or  samples  of  verbal  data  in  order  to  narrow 
the  range  of  interpretation.  To  improve  the  low  inter-rater  agreement  levels  obtained  with  larger 
chunks  from  Experiment  1 ,  communications  from  Experiments  2-4  were  classified  into  smaller 
chunks.  By  Experiment  2,  refinements  in  analysis  methods  including  smaller  verbal  chunks 
resulted  in  more  acceptable  inter-rater  values  ranging  from  .86  for  Factor  to  .99  for  Source. 

The  primary  categories  that  were  used  to  code  each  block  or  chunk  of  verbal  interaction 
by  the  command  group  participants  included: 

■  Fimction:  Plan,  See,  Move,  Strike,  and  BDA. 

■  Source:  Within  Cell,  Cell-Lower,  Cell-Higher,  Higher-Higher. 

■  Factor:  Mission,  Enemy,  Terrain,  Troops,  Time,  and  Civilians. 

■  Type:  Share,  Act,  Direct,  Ask,  Process,  and  Decide. 

■  System:  Platform  break-out  of  air/ground  assets. 

■  Valence:  Positive  versus  negative  status  information. 

■  Command  Considerations:  Cognitive  requirements  of  battle  command. 

The  latter  categories  of  Valence  and  Command  Considerations  were  method  refinements 
introduced  for  Experiments  3  and  4.  Valence  codes  were  added  to  distinguish  communications 
conveying  positive  versus  negative  status  on  accomplishing  basic  C^  functions  and  tasks. 
Command  Considerations  codes  and  classifications,  such  as  situational  awareness,  were  added  to 
relate  battle  command  verbalizations  to  cognitions. 

Human-Computer  Interaction  Analysis 

Human-computer  interactions  (HCI)  were  analyzed  based  on  manual  review  and 
tabulation  of  all  command  group  participant  interactions  with  their  C^  prototypes  based  on  the 
digital  recordings  of  their  C^  interfaces.  Eight  HCI  logs  were  developed  for  each  run  analyzed; 
one  for  each  of  the  dual  displays  at  the  four  participant  workstations.  In  addition,  a  Heads-Up 
Display  (HUD)  ceiling-moimted  in  the  forward  area  of  the  C^  vehicle  (see  Figure  2)  provided  a 
shared  display  to  which  participants  could  toggle  any  of  their  screen  displays  at  any  time.  While 
HCI  logs  captured  participant  use  of  the  HUD,  no  recording  of  the  information  on  the  HUD  was 
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available  to  ARI  researchers.  As  noted,  HCI  analysis  was  not  performed  for  Experiment  1  due  to 
low-resolution  video.  In  addition,  HCI  analysis  was  limited  to  selected  runs  from  Experiments 
2-4  due  to  the  high  analyst  workload  required  in  reviewing  and  tabulating  video  data  from  the 
eight  interfaces  used  by  the  participants. 

HCI  coding  categories  were  based  on  the  C^  functions  of  Plan,  See,  Move,  and  Strike. 
More  detailed  analysis  identified  supporting  sub-functions  and  interactions,  shown  in  Figure  4. 
HCI  coding  was  refined  across  experiments  as  new  features  were  added  to  the  C^  prototype,  such 
as  automated  fires  and  route  generation  routines.  For  Experiments  2-4  respectively,  the  HCI 
codes  were  expanded  from  53  to  83  to  95  different  types  of  interaction.  By  Experiment  3, 
refinements  in  HCI  methods  resulted  in  inter-rater  agreement  of  96%. 


Figure  4.  Evaluative  framework  for  HCI  analysis.  Experiment  4. 

Analysis 

Analytic  methods  and  descriptive  statistics  were  chosen  to  complement  the  exploratory 
versus  controlled  nature  of  the  FCS  C^  experiments.  Comparisons  within  experiments  were 
often  complicated  by  technical  issues,  particularly  for  Experiments  1  and  2.  Between  runs, 
technical  changes  were  sometimes  required  to  reduce  system  crashes  and  to  revise  inadequate 
sensor  models  and  user-identified  shortcomings  in  prototype  features. 
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Comparisons  across  experiments  were  complicated  by  the  program’s  deliberate  focus  on 
new  approaches  to  battle  command  that  stressed  exploration  and  iterative  refinement  of  system 
and  human  parameters  to  meet  emerging  operational  and  doctrinal  requirements  for  FCS  (Unit  of 
Action  Maneuver  Battle  Laboratory,  2003).  For  example,  the  prototype  used  during 
Experiment  1  was  estimated  as  providing  only  about  20%  of  the  full  functionality  envisioned  by 
FCS. 


As  experiments  progressed,  new  technologies  were  added,  and  older  technologies  were 
refined  or  abandoned  based  on  lessons  learned.  System  refinements  and  changes  in  the 
command  group’s  Standing  Operating  Procedures  (SOPs)  resulted  in  repeated  and  progressive 
changes  in  the  allocation  and  performance  of  command  group  functions  and  tasks.  Similarly, 
Unit  Cell  performance  requirements  expanded  from  See/Move  functions  in  Experiment  1  to 
Improved  See/Move/Strike  and  Sustain  functions  by  Experiment  4. 

As  noted,  more  formal  consideration  of  the  Plan  function  was  stressed  by  ARJ  during 
Experiments  3  and  4.  Accordingly,  the  results  reported  below  will  include  the  verbal  and 
human-computer  interactions  of  the  participants  during  the  planning  and  execution  phases  of 
selected  runs.  By  Experiment  4,  this  emphasis  on  planning  resulted  in  more  controlled 
experimental  procedures  during  the  planning  phase,  including  the  requirement  for  a  brief  of  the 
plan  by  the  participant  Unit  Cell  Commander  to  Higher  prior  to  the  execution  phase. 

Results  and  Discussion 

A  small  and  representative  sample  of  the  human  performance  results  from  FCS  C^ 
Experiments  1-4  is  provided  in  this  section.  The  sample  includes  subjective  and  objective 
measures  of  performance  to  help  understand  and  improve  command  group  performance  in  future 
FCS  organizations. 

A  more  complete  account  of  the  human  performance  results  from  each  experiment  is 
provided  in  ARI’s  Interim  Reports  to  PM  FCS  C^.  For  Experiment  3,  more  complete  results  on 
human  performance  are  also  provided  in  Lickteig,  Sanders,  Durlach,  and  Carnahan,  in 
preparation.  Operational  results,  including  measures  of  performance  and  effectiveness,  were 
documented  by  other  researchers  and  are  also  available  from  PM  FCS  C^. 

Subjective  Results 

A  select  set  of  subjective  results  from  the  FCS  C^  experiments  is  provided  in  this  section 
under  the  topics  of  workload,  human  performance,  system  performance,  and  training.  A  sample 
of  participant  comments  from  the  In-Place  AARs  is  included  to  present  an  “in  their  own  words” 
account  of  participant  feedback,  and  to  indicate  the  range  of  subjective  information  on  human 
performance  provided  to  and  available  from  PM  FCS  C^. 

Workload 

Workload  was  targeted  as  a  key  human  performance  concern  given  the  FCS  goal  of 
reducing  the  command  group  persotmel  assigned  to  a  small  unit  with  manned  and  predominantly 
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robotic  elements.  Therefore,  the  Too  High  level  of  run  complexity  was  designed  to  test  human- 
system  limitations  and  more  precisely  identify  workload  and  performance  thresholds  of  the 
proposed  command  group  structure  and  Unit  Cell  organization. 

Figure  5  depicts  exemplar  results  on  subjective  ratings  of  workload  from  Experiment  3 
by  duty  position  and  run  complexity.  Mean  workload  values  were  calculated  by  averaging 
participant  ratings  across  the  scale  items  on  the  mental,  physical,  temporal,  effort,  and  frustration 
dimensions  of  work. 


Figure  5  indicates  an  increase  in  command  group  workload  particularly  during  the  Too 
High  runs.  The  increase  was  most  pronounced  for  the  duty  positions  of  Battlespace  Manager  and 
Information  Manager  during  more  complex  runs.  Workload  ratings  by  complexity  from 
Experiments  1  and  2  showed  a  similar  pattern.  Recall,  complexity  was  not  varied  during 
Experiment  4. 
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Figure  5.  Mean  workload  ratings  by  duty  position  and  run  complexity.  Experiment  3. 

(Scale:  0  =  Very  Low,  100  =  Very  High) 

Across  the  four  experiments,  a  comparison  of  workload  ratings  by  duty  position  is  shown 
in  Figure  6.  When  averaged  across  duty  positions  and  run  complexity  for  Experiments  1  -3,  the 
overall  mean  workload  ratings  were:  58, 61, 49,  and  69  for  Experiments  1-4,  respectively. 
However,  caution  is  exercised  and  urged  in  efforts  to  explain  apparent  differences  in  workload 
ratings  between  experiments. 

In  sum,  the  subjective  results  suggest  that  the  participants  experienced  moderate  to  high 
levels  of  workload  in  the  FCS  command  group  and  the  Unit  Cell  organization  as  examined. 
Subjective  workload  was  not  uniform  across  duty  positions.  The  Battlespace  Manager  and 
particularly  the  Information  Manager  generally  reported  higher  workload  across  experiments. 
Overall,  the  results  suggest  that  during  more  complex  runs  the  participants  were  relatively  busy, 
if  not  stressed,  by  the  workload  required  to  command  and  control  Unit  Cell,  particularly  the  array 
of  unmanned  ground  and  air  assets  proposed  for  smaller  FCS  units. 
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Figiire  6.  Mean  workload  ratings  by  duty  position,  Experiment  3. 

(Scale:  0  =  Very  Low,  100  =  Very  High) 

Human  Performance 

Performance  success  is  almost  always  a  key  concern.  From  a  human  performance 
perspective,  ARI  focused  on  participant  ratings  of  successful  performance  relative  to  their  duty 
position  requirements.  Figure  7  summarizes  ratings  of  performance  success  from  Experiment  3 
by  duty  position  and  run  complexity.  Results  indicate  a  relatively  sharp  decline  in  performance 
success  at  the  Too  High  level  of  complexity  across  the  four  command  group  participants. 

Particularly,  the  Information  Manager’s  low  ratings  may  indicate  serious  hmnan  system 
integration  issues.  Issues  identified  included  the  difficulty  of  controlling  the  Unit  Cell’s 
unmanned  aerial  vehicles  (UAVs)  and  analyzing  the  larger  amount  of  sensor  images  in  the  Too 
High  condition.  Performance  success  ratings  by  run  complexity  were  similar  for  Experiments  1 
and  2  and  were  not  available  for  Experiment  4. 
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Figure  7.  Mean  performance  success  ratings  by  duty  position  and  run  complexity.  Experiment  3. 
(Scale:  0  =  Failure,  100  =  Perfect) 
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System  Performance 

More  precise  subjective  measures  were  developed  to  assess  the  effectiveness  of  key  new 
features  in  the  prototype,  as  they  were  progressively  introduced  into  the  experiments.  The 
largest  suite  of  new  features  was  inserted  in  Experiment  3,  as  suggested  by  an  increase  in  HCI 
codes  from  53  to  83  between  Experiments  2  and  3.  The  13  new  features  added  for  Experiment  3 
are  identified  in  Figure  8. 

Figure  8  depicts  mean  ratings  of  feature  effectiveness  averaged  across  duty  position  and 
selected  runs  (Runs  5  and  10)  from  Experiment  3.  Overall,  most  of  the  new  features  received 
positive  ratings.  For  example,  “Very  Effective”  ratings  were  given  to  two  Attack  Guidance 
Matrix  (AGM)  features  that  automated  some  time-consuming  and  time-limited  tasks  under  the 
Strike  fiinction. 


In  contrast,  a  See-related  feature  called  HTR  Viewer  (Human  Target  Recognition)  and  a 
Move  fiinction  feature  called  Group  Tasking  were  rated  “Ineffective”  to  “Neutral.”  Notably, 
questionnaire  results  on  new  feature  effectiveness  included  written  conunents  obtained  from* 
each  participant  that  were  routinely  used  to  help  assess  and  refine  the  features  of  the  prototype 

for  fiiture  efforts. 
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New  Features  for  Experiment  3 


Figure  8.  Mean  ratings  of  new  features  effectiveness.  Experiment  3. 
(Scale:  1  =  Very  Ineffective,  5  =  Very  Effective) 


Training 


Results  on  training  are  based  on  participant  feedback  on  the  training  questionnaires  and 
author  observations.  Overall,  training  improved  substantially  over  the  course  of  experiments 
including  development  and  refinement  of  A  User’s  Manual,  primarily  for  individual  training. 
However,  shortcomings  in  training  were  observed  and  confirmed  by  participants  that  impacted 
performance,  and  may  impact  future  performance  in  FCS  units. 
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Notably,  participants  were  more  critical  about  training  on  the  Post-Run  versus  Pre-Run 
training  survey  from  Experiment  4.  This  finding  seems  a  reminder  that  often  “you  don’t  know 
what  you  don’t  know”  imtil  you  do  the  job,  or  do  the  research.  Even  after  Experiment  4,  the 
command  group  participants  who  had  completed  40+  runs  across  experiments  expressed  strong 
coneems  about  training  and  their  ability  to  fully  exploit  Unit  Cell  capabilities: 

■  Need  more  hands-on  in  taetieal  scenarios  or  vignettes,  less  lecture.  Putting  the  lesson  in 
a  taetieal  situation  really  lends  eredence  to  the  function  you  are  teaching  and 
demonstrates  why  you  want  to  learn  it  as  well  as  how  it  is  best  employed. 

■  Training  should  be  more  dedicated  to  aetual  employment  techniques.  We  could  have 
used  more  time  integrating  as  a  team. 

■  We  get  more  functional  every  run,  what  we  need  are  a  couple  runs  designed  for  us,  not  to 

be  critiqued  (i.e.,  in  AAR),  but  for  us  to  re-establish  SOPs....  Not  an  easy  task . 

Overall,  the  training  results  stressed  the  need  for  progressive,  structured  training  with 
simulation-based  feedback  during  and  after  performance.  These  and  related  training  issues  will 
be  addressed  in  subsequent  sections  of  this  report  titled  “Status  Report  on  Research  Objectives” 
and  "Conclusions.” 

Jn- Place  AAR 

A  sample  of  comments  provided  by  each  of  the  four  participants  during  the  In-Place 
AAR  is  provided  below.  Overall,  the  data  provided  to  the  PM  PCS  C^  generally  included  all 
participant  comments  reeorded  by  duty  position  during  the  In-Place  AARs.  Sample  eomments 
below  arc  from  three  runs  during  Experiment  4.  Comments  are  direetly  from  the  participants  by 
run  and  dutj’  position  with  minor  editing,  including  acronyms  spelled  out  for  clarity. 

Batticspace  Manager  Comments: 

•  Run  2.  What  went  well  was  that  we  did  one  group  moimt,  no  group  follow,  no  auto  route; 
all  manual.  Zoomed  down  to  identify  navigable  trails,  auto  route  would  not  have  worked. 
Good  UAV  coordination  painting  the  Valley  of  Death.  Flanked  him  and  turned  him  out 
of  position. 

■  Run  6.  There  were  more  targets  than  we  could  service.  Almost  impossible  to 
diseriminate  between  live  and  dead  targets.  The  enemy  was  so  close  he  moved  under 
rounds.  Blue  could  only  maneuver  North  and  South  as  we  had  our  back  against  the  wall. 

■  Run  8.  What  went  wrong,  one  vehicle  got  stuek,  and  I  really  don’t  know  why.  Turning 
range  fans  off  and  on  worked  well.  Group  follow  worked,  generally  happy  with  the  run. 

Information  Manager  Comments: 

■  Run  2.  Had  one  seetor  to  focus  on,  the  South  main  avenue.  Had  no  pictures  in  the  Miero 
UAV  (Unmanned  Aerial  Vehicle)  quad  view  images. 

■  Run  6.  I  agree  with  the  Battlespace  Manager.  There  were  too  many  images,  too  much 
clutter,  smoking,  confusing  images,  dual  images.  There  were  a  lot  of  enemy  assets  up 
front  that  knoeked  out  our  assets.  Couldn’t  tell  whether  targets  were  dead  or  alive. 
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■  Run  8.  Auto  functions  worked,  namely  Auto  Recon  and  Recon  Location.  What  went  bad 
was  that  Shadow  died  flying  low.  Micro  UAVs  worked  well  to  confirm  the  target  set. 

We  could  have  gotten  through  to  the  objective. 

Effects  Manager  Comments: 

■  Run  2.  Changed  things  today,  started  with  blank  AGM  (Attack  Guidance  Matrix)  so  it 
didn’t  interfere  with  manual  fires.  The  AGM — best  ever  dealing  with  all  the  targets  that 
popped  up — ^saved  time  and  allowed  for  me  to  focus  on  the  mid-range  battle. 

■  Run  6.  Not  too  many  targets  for  AGM  to  engage,  the  problem  was  that  there  were  too 
many  images.  The  tradeoff  is  that  when  there  are  a  lot  of  images  there  is  no  time  for 
image  assessment.  There  is  not  a  problem  with  engaging  targets.  The  test  managers 
have  got  to  extend  the  terrain  box;  we  would  have  never  put  ourselves  into  that  position 
with  Red  directly  in  front. 

■  Run  8.  A  bit  more  ammunition  consumption  than  earlier  runs.  The  commander  used  me 
like  a  lawn  mower. 

Unit  Cell  Commander  Comments 

■  Run  2.  What  went  right  was  the  early  read,  and  Blue  Higher  coordination  for  A-1 60 
sensors  support.  Maneuver  went  good:  go  South  which  was  his  weakness,  single 
envelopment  into  his  move.  Best  run  across  all  experiments.  We  got  a  lot  out  of 
Shadow,  tradeoff  worth  it. 

■  Run  6.  We  cannot  discriminate  targets,  so  we  miss  some,  and  those  engaged  are  not  all 
destroyed,  an  effect  of  munitions  used,  so  BDA  is  challenging.  There  was  no  room  in  the 
box  to  maneuver.  Can’t  discriminate  between  dead  and  live  targets. 

■  Run  8.  Good  read  early.  Did  not  have  great  read  on  enemy’s  air  defense.  Overall  good 
read,  the  plan  allowed  us  to  destroy  him  in  detail. 

Objective  Results 

A  small  representative  sample  of  objective  results  is  provided  in  this  section  based  on  the 
command  group  participants’  verbal  and  human-computer  interactions. 

Verbal  Interaction 

Results  on  the  verbal  interactions  of  command  group  participants  across  Experiments  1  -4 
are  organized  by  C^  function,  source,  amount,  and  valence. 

Function.  What  is  the  function  of  communication?  The  analysis  of  verbal  interaction 
addressed  this  question  by  relating  participant  communications  to  the  basic  C^  functions  of  Plan, 
See,  Move,  Strike,  and  BDA.  Figure  9  provides  exemplar  data  on  percentage  of  verbalizations 
across  duty  positions  by  C^  function  during  Run  10  from  Experiment  4. 

Verbalizations  related  to  the  See  funetion  were  clearly  the  most  frequent,  about  30%  of 
all  participant  verbal  communications.  This  result  appears  to  underscore  the  command  group’s 
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extensive  effort  to  control  and  interpret  input  from  a  set  of  manned  and  particularly  unmanned 
networked  sensors  to  maintain  an  accurate  battlefield  “picture.”  Strike  and  BDA  were  also  main 
topics,  reflecting  the  command  group’s  focus  on  destroying  and  clearing  enemy  threats  to  their 
Unit  Cell  and  follow-on  forces. 
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Figure  9.  Percent  of  verbalization  by  function,  Experiment  4. 


Source.  Who’s  talking?  A  key  to  understanding  command  and  control  verbalizations  is 
tracking  the  source.  Initial  analysis  found  that  across  experiments  approximately  90%  of  all 
verbalizations  occurred  within  the  command  group.  Remaining  verbalizations  were  primarily 
between  the  command  group  and  surrogate  higher  and  lower  elements  role-played  by  support 
personnel.  Subsequent  analysis  more  closely  examined  the  source  of  command  group 
verbalizations.  Figure  10  depicts  the  mean  percentage  of  verbalizations  by  duty  position  across 
experiments.  The  Commander  was  clearly  the  dominant  source  of  the  command  group’s 
verbalizations.  Across  the  four  experiments,  the  mean  percentage  of  communications  initiated 
by  participant  was:  53%  for  the  Commander,  19%  for  die  Information  and  Battle  Space 
Managers,  and  8%  for  the  Effects  Manager. 
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Figure  10.  Percent  of  verbalization  by  duty  position  across  experiments. 

Amount.  An  interesting  aspect  of  the  verbal  data  is  that  verbal  interaction  by  the 
command  group  was  an  almost  continuous  activity.  Breaks  in  verbalization,  or  silent  time, 
averaged  7%  of  total  run  time  across  experiments.  The  finding  that  verbalizations  occurred  93% 
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of  the  time  during  the  execution  phase  appears  noteworthy.  The  pattern  of  steady  conversation 
occurred  notwiths^ding  the  participants’  common  access  to  a  visually  rich  and  timely  depiction 
of  the  battlefield  situation  on  their  displays.  It  also  occurred  despite  numerous  ongoing 
human-computer  interactions  to  see  the  battlefield  and  control  robotic  entities  (see  HCI  results 
below).  Such  results  highlight  the  very  important  role  of  verbal  communication  for  battle 
command  collaboration,  even  with  advanced  information  technologies. 


Valence.  A  perceived  shortcoming  in  verbal  analysis  after  Experiment  2  was  a  failure  to 
account  for  positive  versus  negative  information.  Evaluative  meaning  is  conveyed,  or  sought,  in 
nearly  all  communication.  For  command  and  control,  evaluative  information  should  directly 
support  the  decision  making  process.  Valence  codes  were  added  for  Experiments  3  and  4, 
therefore,  to  discern  what  verbalizations  conveyed  on  the  status  of  accomplishing  functions 

and  tasks. 


Figure  1 1  shows  mean  verbalization  percentages  by  valence  from  Run  1 0,  Experiment  3. 
This  figure  provides  valence  data  by  function  and  time,  namely  run  quartiles.  Overall,  most 
verbid  communications  conveyed  positive  information  on  the  status  of  accomplishing  ’ 

functions  and  tasks.  However,  communications  related  to  Move  and  BDA  functions  were  more 
negative.  Negative  verbalizations  were  especially  useful  in  identifying  and  addressing 
shortcomings  in  the  prototype,  such  as  difficulty  with  Group  Follow  tools  and  BDA. 
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Figure  11.  Percent  of  verbalization  by  function,  valence,  and  time  (quartiles).  Experiment  3 
Human-Computer  Interaction 


Results  on  the  human-computer  interactions  of  command  group  participants  across 
Experiments  2-4  are  organized  by  function,  duty  position,  workload,  and  time. 

(^Function.  FCS  concepts,  particularly  for  command  and  control,  rely  heavily  on  an 
alliance  of  humans  and  computers.  To  better  understand  and  build  the  alliance,  an  in-depth  HCI 
analysis  was  performed  on  selected  runs.  The  analysis  tabulated  and  related  each  and  every 
human-computer  interaction  by  the  command  group  to  the  functions  and  sub-functions  that 
were  previously  identified  (see  Figure  4).  The  HCI  codes  and  analysis  methods  were  expanded 
and  revised  to  address  the  increase  from  53  to  95  different  types  of  interactions,  as  features  were 
added  or  modified  between  Experiments  2-4. 
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Table  4  provides  an  exemplar  result  of  the  HCI  frequency  data  obtained  for  the  PCS  C 
experiments.  Notably,  the  data  provided  in  Table  4  is  from  the  planning  phase  of  a  run  from 
Experiment  3.  Data  from  a  planning  phase  is  provided  to  indicate  the  scope  of  results  provided 
to  and  available  from  PM  PCS  C^.  The  planning  data  will  also  be  used  here  to  help  describe 
some  of  the  key  prototype  features  and  related  human-computer  interactions  that  supported 
the  Plan  fiinction. 

Table  4  provides  a  detailed  account  of  the  499  total  human-computer  interactions 
performed  and  tabulated  by  duty  position  during  the  planning  phase  of  Run  10,  Experiment  3. 
Note,  only  9  of  17  sub-functions  and  26  of  the  83  types  of  interaction  available  on  the 
prototype  for  Experiment  3  were  actually  performed  during  the  planning  phase  for  this  run. 

A  graphic  summary  of  the  same  planning  data  is  provided  in  Pigure  12  that  depicts  HCI 
frequency  by  function.  During  this  relatively  typical  planning  phase  the  overall  percentage  of 
command  group  interactions  across  duty  positions  by  function  was:  Plan  (36%),  Move 
(1 1%),  See  (51%),  and  Strike  (2%). 

An  interaction  particularly  indicative  of  the  prototype’s  features  in  support  of  the  Plan 

function  was  repeated  (22  times)  use  of  the  Rehearse  Plan  animation  tool.  The  Rehearse  Plan 
tool  pictorially  depicted  the  combined  movement  of  ground  and  air  assets  across  the  battlefield, 
as  planned.  This  animation  tool  was  used  to  coordinate  plans  within  the  command  group  and 
especially  to  provide  dynamic  back-briefs  of  the  plan  by  the  Commander  to  higher  echelons. 

See-related  interactions  during  planning  primarily  involved  using  the  Query  tool  by 
moving  the  cursor  over  Enemy  (65  times)  and  Friendly  (44  times)  icons  to  access  information 
such  as  vehicle  type  and  position,  and  the  Plot  Intervisibility  tool  (27  times)  to  view  cover  and 
concealment  during  the  planning  of  ground  and  air  asset  movement.  Other  interactions  in 
support  of  the  Plan  function  primarily  involved  designating  vehicle  icon  locations  and  routes, 
and  inserting  graphic  control  measures. 

Duty  Position.  Figure  13  provides  an  exemplar  summary  of  HCI  frequency  by  function, 
duty  position,  and  display  during  the  subsequent  execution  phs&Q  of  Run  10,  Experiment  3. 
Overall,  the  1,044  human-computer  interactions  performed  and  tabulated  in  the  execution  phase 
were  associated  with  the  following  functions:  Move  (5%),  See  (75%),  and  Strike  (20%). 

In  general,  interactions  in  support  of  the  See  function  were  most  frequent  in  the  execution 
phase,  as  in  planning.  Also,  the  fact  that  almost  no  Plan-related  interactions  were  observed 
during  execution  may  reflect  limitations  in  system  design  and  analysis  methods  that  should  be 
overcome  in  future  efforts,  and  that  planning  was  to  some  extent  curtailed  by  the  deliberate 
practice  design. 
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Table  4 


Frequency  of  HCI  by  Duty  Position  in  Planning  Phase,  Experiment  3 


Conunand  and  Control 

Duty  Position 

Function 

Commander 

Battlespace 

Information 

1  Effects  Manager 

Sub-function 

Interaction 

Freq. 

% 

Manager 

Freq.  % 

Manager 

Freq.  % 

Freq. 

% 

Plan 

76 

63.3 

42 

25.8 

60 

35.7 

1 

2.1 

Create  Mission/Course  of 

Action  (COA) 

76 

63.3 

42 

25.8 

60 

35.7 

1 

2.1 

Create  Overlay  Graphics 

— 

— 

1 

.6 

Place  Platforms  on  Map 

1 

38 

— 

_ _ 

14 

8.3 

Rehearse  the  Plan 

4 

3.4 

10 

6.1 

8 

4.8 

Move  Icons  (Drag/Drop) 

70 

58.3 

21 

12.9 

28 

1.7 

1 

2.1 

Modify  Overlay  Graphics 

1 

.8 

10 

6.1 

10 

6.0 

Move 

— 

25 

15.3 

26 

15.5 

1 

2.1 

Move  Ground  Assets 

— 

— 

24 

14.7 

5 

3.0 

1 

2  1 

Create  Routes 

— 

— 

7 

4.3 

_ 

Edit  Existing  Route 

— 

— 

4 

2.5 

Delete  all  Tasks 

— 

— 

8 

4.9 

5 

3.0 

1 

2.1 

Create  Overwatch  Task 

— 

1 

.6 

Generate  Route 

— 

— 

4 

2.5 

_ 

Move  Air  Assets 

— 

_ 

21 

12.5 

Recon  an  Area 

— 

— 

— 

21 

12.5 

Group  Follow 

— 

— 

1 

.6 

^ _ 

Create  Ground  Follow 

— 

I 

.6 

See 

44 

36.7 

96 

58.9 

82 

48.8 

34 

7.08 

Manipulate  M^ 

10 

8.3 

10 

6.2 

15 

8.9 

18 

37.5 

Zoom  Map 

8 

6.7 

6 

3.7 

12 

7.1 

7 

14.6 

Scroll  Map 

2 

1.6 

4 

2.5 

3 

1.8 

11 

22.9 

Use  Visualization  Aids 

6 

5.0 

47 

28.8 

14 

83 

2 

4.2 

Plot  Intervisibility 

— 

— 

27 

16.6 

Display  on  Heads  Up 

— 

— 

2 

1.2 

,  , 

Select/Change  Windows 

6 

5.0 

11 

6.7 

14 

8.3 

2 

4.2 

Change  Graphic  Control  Measure 
(GCM)  Settings 

— 

7 

4.3 

— 

Display  Sensor  Data 

26 

21.7 

39 

23.9 

44 

26.2 

12 

25.0 

Query  Enemy 

17 

14.2 

10 

6.1 

31 

18.5 

7 

14.6 

Query  Friendly 

9 

7.5 

27 

16.6 

13 

7.7 

5 

10.4 

Toggle  Sensor  Fans 

— 

2 

1.2 

Recognize  Targets 

2 

1.7 

_ 

9 

5.4 

2 

4.2 

Change  Icon  Type 

2 

1.7 

— 

... 

9 

5.4 

2 

4.2 

Strike 

— 

— 

— 

— 

12 

25.0 

Fire  a  Weapon 

— 

— 

— 

— 

_ 

— 

12 

25.0 

Fire  LAM 

— 

— 

— 

— 

_ _ 

— 

3 

6.3 

Fire  Precision  Attack  Missile 
(PAM) 

— 

~ 

— 

6 

12.5 

Fire  LOS 

— 

1 

2.1 

Delete  Fire  Tasks 

— 

— 

— 

r, 

2 

4.2 

Total 

120 

100 

163 

100 

168 

100 

48 

100 

28 


Plan  Monas  See  Strike 

Function 

Figure  12.  Frequency  of  interaction  by  function  during  planning  phase.  Experiment  3. 

Duty  Position.  Figure  13  provides  an  exemplar  summary  of  HCI  frequency  by  function, 
duty  position,  and  display  during  the  subsequent  execution  phase  of  Run  10,  Experiment  3. 
Overall,  the  1,044  human-computer  interactions  performed  and  tabulated  in  the  execution  phase 
were  associated  with  the  following  functions:  Move  (5%),  See  (75%),  and  Strike  (20%). 

Overall,  interactions  in  support  of  the  See  function  were  most  frequent  in  the  execution  phase,  as 
in  planning.  Also,  the  fact  that  almost  no  Plan-related  interactions  were  observed  during 
execution  may  reflect  limitations  in  system  design  and  analysis  methods  that  should  be  overcome 
in  future  efforts,  and  that  planning  was  to  some  extent  curtailed  by  the  deliberate  practice  design. 


Figure  13.  Frequency  of  interaction  by  function,  duty  position  and  display.  Experiment  3. 

A  brief  description  of  some  key  command  group  interactions  during  the  execution  phase 
is  provided  below  by  duty  position  to  help  clarify  the  type  and  range  of  potential  interactions 
required  by  a  command  group  in  smaller  FCS  units: 

■  Commander  interactions  were  almost  exclusively  (96%)  See-related  interactions  during 
execution.  Majority  of  Commander’s  interactions  were  Display  Sensor  Data  tasks  to 
access  Enemy  data,  and  use  of  the  Unit  Viewer  to  maintain  situational  awareness  and 
status  data  on  key  Friendly  and  Enemy  assets. 
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■  Battlespace  Manager  interactions  by  &  function  were  primarily  See  (60%),  Strike 
(29%),  and  Move  (9%)  related.  Strike  interactions  were  mainly  Target  Designate  (41 
times)  and  Fire  Weapon  (46  times).  Move-related  interactions  included  navigating 
ground  assets  by  deleting  previously  entered  planned  routes  and  entering  new  routes,  as 
well  as  starting,  halting,  and  resuming  movement  patterns. 

■  Information  Manager  interactions  were  primarily  See  (89%)  and  Move  (8%)  related 
actions.  Particularly,  the  See-related  interactions  included  repeated  call-up  and  review  of 
sensor  images  in  support  of  Human  Target  Recognition  (HTR)  and  BDA.  Movement 

interactions  included  routing  and  positioning  air  assets,  mainly  unmanned  aerial  vehicles 
(UAVs). 

■  Effects  Manager  interactions  primarily  supported  See  (64%)  and  Strike  (35.4%) 
functions.  Strike  actions  included  Target  Designate  (28  times).  Fire  Weapon  (32  times), 
and  Monitor  LAM^AM)  (Loitering  Attack  Missile  and  Precision  Attack  Missile) 
engagements  (24  times).  See-related  interactions  included  numerous  Manipulate  Map 
(91  times)  and  Display  Sensor  Data  (62  times)  actions. 

Figure  13  also  provides  HCI  frequency  by  display,  or  interface.  As  the  participants’ 
dual  displays  were  equivalent  and  redundant.  Figure  1 3  indicates  how  each  participant  elected  to 
perform  interactions  by  display.  Display  preferences  were  clearly  not  uniform:  left  display  used 
primarily  by  the  Effects  Manager,  right  display  by  the  Battlespace  Manager,  and  fairly  balanced 
left  and  right  display  use  by  the  Commander  and  the  Information  Manager.  Such  results  raise 
interesting  questions  about  individual  differences,  duty  position  differences,  and  interface  design 
that  future  efforts  might  address  to  improve  human  system  integration. 

Workload.  Figure  14  provides  a  more  comprehensive,  across  Experiments  2-4,  look  at 
HCI  frequency  by  duty  position.  Of  course,  the  HCI  data  represents  only  a  portion  of  each 
participant’s  overall  duty  position  activities.  The  results  indicate  clear  differences  between 
Commander  and  subordinate  interaction  patterns. 
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Figure  14.  Frequency  of  interaction  by  duty  position  and  experiment. 


For  example,  the  Commander’s  relatively  low  level  of  interaction,  aimed  at  seeing  the 
battlefield,  appeared  more  consistent  with  a  “thinker”  versus  “doer”  role  for  battle  command. 
Subordinate  interactions  were  much  higher,  especially  by  the  Information  and  Battle  Space 
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Managers.  Subordinates  averaged  four  (4)  interactions  per  minute  across  Experiments  2-4. 
However,  peak  performance  by  subordinates  during  more  intense  segments  of  a  run  was  9-12 
interactions  per  minute. 

A  more  basic  question  raised  by  the  results  in  Figure  14  might  be  what  is  the  impact  of 
automation  on  workload?  Refinements  in  the  prototype  across  experiments,  particularly  the 
addition  of  new  features,  were  expected  to  at  least  partially  automate  many  tasks  and  functions 
for  command  and  control.  However,  Figure  14  discloses  a  sharp  and  uniform  drop  in  HCI 
frequency  across  the  command  group  only  for  Experiment  3.  Notably,  participants’  subjective 
ratings  of  overall  workload,  previously  shown  in  Figure  6,  disclosed  a  very  similar  pattern  across 
Experiments  2-4,  with  respective  mean  workload  values  of  61, 49,  and  69. 

From  a  measurement  perspective,  such  results  provide  convergent  validity  across 
objective  and  subjective  measurement  methods.  From  a  causal  perspective,  the  impact  of 
automation  on  workload  is  less  certain  and  arguably  more  complicated.  One  explanation  for  the 
Experiment  4  increase  in  interaction  and  workload  may  be  that  performance  requirements  were 
increased  by  programmatic  expectations  about  baseline  performance  capabilities,  including 
repeated  requests  for  faster  mission/run  completion. 

Time.  An  example  of  the  observable  “trends”  in  HCI  performance  by  duty  position  and 
time  is  provided  in  Figure  15.  This  figure  depicts  interaction  frequency  in  successive  ten-minute 
intervals,  or  segments,  of  execution  time  during  Run  10  of  Experiment  3.  Note  the  nm’s  final 
time  segment  (80-84  minutes)  was  deleted  to  avoid  comparison  across  different  time  intervals. 
Interesting  differences  by  time  and  duty  position  are  clearly  evident. 


■  -  Commander 

— Battlespace 
•  •k-  -  Information 
— Effects 


10  Minute  Intervals 


Figure  15.  Frequency  of  interaction  by  duty  position  and  time,  Experiment  3. 

Complementary  narrations  were  also  developed  by  the  analysts  to  more  fully  describe 
and  summarize  the  interactions  performed  by  each  participant  during  each  10-minute  block  of 
each  run  analyzed.  In  particular,  results  such  as  the  frequency  by  time  data  indicated  in  Figure 
15  should  help  to  identify  and  understand  variations  and  collaboration  in  the  performance  of 
future  command  groups. 


31 


Automated  Measures 


The  requirement  for  automated  measures  of  performance  to  support  training  and 
evaluation  (Unit  of  Action  Maneuver  Battle  Laboratory,  2003)  is  underlined  by  the  HCI  analyst 
workload  for  PCS  experiments.  For  example,  manual  reduction  of  the  HCI  data  for  Run  10 
from  Experiment  3  required  16  analyst  workdays  to  identify  and  tabulate  the  499  human- 
computer  interactions  during  planning  and  the  1 ,044  interactions  during  execution. 

Overall,  results  on  the  development  and  validation  of  automated  measures  are  promising 
but  quite  meager.  Only  3  of  the  23  automated  measures  requested  by  ARI  and  listed  in  Table  3 
were  actually  developed.  Only  2  of  those  3  measures  were  validated,  at  least  partially.  The  three 
automated  measures  developed  were:  Alert  Acknowledged,  Image  RequestedA^iewed,  and 
Create  Ground/Air  Route. 

Efforts  to  validate  the  measure  Alert  Acknowledged  were  not  successful.  As  indicated 
in  Table  3,  the  purpose  of  this  measure  was  to  capture  the  time  to  respond  to  an  alert  by  turning  it 
off  However,  the  parsed  logger  files  provided  to  ARI  for  Alert  Acknowledged  included 
unnecessary  and  confusing  information. 

For  example,  time  data  included  two,  sometimes  three,  different  time  stamps  that  could 
not  be  readily  distinguished  or  matched  to  the  run  times  available  with  the  video  recordings  used 
for  manual  data  reduction.  Images  associated  with  some  alerts  were  identified  on  the  log  with 
system  file  names  (e.g.  \\ucl-images\images\ir_10.ntf )  versus  the  C^  prototype  image  names 
(e.g.,  Garm  23)  available  with  the  video  recordings.  Moreover,  the  C^  prototype  provides  em 
array  of  user  options  for  setting  and  responding  to  alerts  that  complicates  measure  definition  and 
extraction. 

Efforts  to  validate  the  measures  Image  RequestedA^iewed  and  Create  Ground/ Air  Route 
were  fairly  successful.  Table  5  summarizes  the  validation  results  with  a  comparison  of  the 
frequencies  tabulated  for  these  two  measures  by  automated  and  manual  measurement  methods. 
There  were  no  discrepancies  in  the  number  of  images  requested/viewed  by  the  Commander  and 
Effects  Manager  who  accessed  their  images  from  the  Battlefield  Assistant  in  the  Alert  Window. 

However,  there  were  notable  discrepancies  in  this  same  measure  for  the  Battlespace  and 
Information  Manager.  These  discrepancies  were  due  to  shortcomings  in  the  measure  that  did  not 
count  images  viewed  by  clicking  Chaser  round  icons  that  only  appeared  on  the  Map  Window. 

Similarly,  there  were  no  discrepancies  in  the  measured  number  of  ground  routes  created 
by  the  Battlespace  Manager.  There  was  only  a  small  discrepancy  (19  versus  18)  in  the  measured 
number  of  air  routes  created  by  the.  Information  Manager.  This  appeared  to  be  due  to  the  fact  that 
manual  measures  tabulated  only  the  number  of  air  routes  actually  generated,  rather  than  the 
number  of  air  routes  attempted  but  not  suecessfully  completed. 
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Table  5 


Comparison  of  Automated  Measures  to  Manual  HCI  Data  Reduction 


Duty  Position 

Prequency  Accoimted  Por 

Measured  task 

Automated  Measures 

Manual  HCI  Analysis 

Commander 

Image  RequestedA^iewed 

9 

9 

Battlespace  Manager 

Create  Groimd  Route 

11 

11 

Image  RequestedA^iewed 

2 

24 

Information  Manager 

Create  Air  Route 

19 

18 

Image  RequestedA^iewed 

62 

72 

Effects 

Image  RequestedA^iewed 

4 

4 

In  summary,  the  discrepancies  identified  in  Table  S  for  the  two  automated  measures 
actually  developed  are  notable  but  should  be  readily  resolvable.  Development  and  validation  of 
automated  measures  is  almost  always  an  iterative  process  of  collaboration  between  behavioral 
and  technical  experts.  However,  the  discrepancies  for  Alert  Acknowledged  and  especially  the 
failure  to  develop  the  additional  automated  measures  imderscore  the  basically  unmet  requirement 
for  automated  measures  of  performance  (Unit  of  Action  Maneuver  Battle  Laboratory,  2003). 

Status  Report  on  Research  Objectives 

This  section  provides  a  brief  recap  or  summary  assessment  on  the  research  objectives  that 
directed  ARI’s  involvement  in  the  PCS  research  program,  as  described  in  the  earlier  section 
titled  “Human  Performance  Research  Objectives.” 

Initiate  Empirical  Database 

The  research  reported  here  was  successful  in  initiating  the  development  of  an  empirical 
database  on  future  command  group  performance  for  PCS.  The  methods  and  results  provide 
reliable  and  empirical  groundwork,  if  not  benchmarks,  on  the  human  performance  requirements 
for  PCS  command  groups  and  new  approaches  to  command  and  control.  This  report  emphasizes 
the  scope  and  depth  of  empirical  data  on  human  performance  provided  to  and  available  from  the 
PM  PCS  cl 

Prankly,  ARI  researchers  and  managers  were  at  times  concerned  the  methods  used  to 
measure  human  performance  were  too  detailed  and  comprehensive,  including  practically  every 
command  group  utterance  and  interaction  during  the  planning  and  execution  of  selected  runs. 

However,  new  and  largely  uncharted  approaches  to  command  and  control  were  at  issue. 
And  ARI  was  committed  to  better  understanding  and  documenting  how  future  command  groups 
envisioned  by  PCS  might  perform.  Similarly  detailed  assessments  of  verbal  and  computer 
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interaction  by  command  and  control  performers  were  made  by  Navy  researchers  examining  the 
impact  of  advanced  technology  on  decision  making  under  stress  (e.g.,  Morrison,  Kelly,  Moore 
&  Hutchins,  1 998). 

In  sum,  a  relatively  comprehensive  and  empirical  database  on  future  command  group 
performance  for  FCS  was  initiated.  The  subjective  measures  about  performance  addressed 
multiple  research  issues  including;  training,  skill  proficiency,  workload,  performance  success, 
teamwork  skills,  decision  making,  function  and  task  allocations,  prototype  effectiveness,  and  ’ 
human-system  integration.  The  objective  measures  of  performance  related  relatively  micro 
behaviors,  particularly  verbal  and  human-computer  interaction,  to  relatively  macro  level 
command  and  control  functions  including  Plan,  See,  Move,  and  Strike.  Measurement  methods 
and  data  on  human  performance  were  documented  to  support  future  research  in  a  series  of 
Interim  Reports  available  from  PM  FCS  and  ARI  reports  (Carnahan,  Lickteig,  Sanders,  and 
Durlach,  in  preparation;  Lickteig,  Sanders,  Durlach,  &  Carnahan,  in  preparation). 

Improve  Research  Environment 

^The  research  reported  here  was  at  least  partially  successful  in  iteratively  improving  the 
^  research  environment  for  latter  experiments.  Shortcomings  in  human-human  interaction 
and  human-computer  interaction  in  particular  were  routinely  identified  and  documented  by  ARI 
during  and  after  each  experiment.  Refinements  were  made  to  redress  many,  certainly  not  all,  of 
the  shortcomings  identified  in  subsequent  experiments  to  improve  system  design,  experimental 
design,  training,  and  methods  for  measuring  human  performance. 

At  times,  improvements  were  actually  implemented  within  an  ongoing  experiment  to 
o\  ercomc  unacceptable  problems  including  user-identified  shortcomings  in  the  functions  and 
features  of  the  C‘  prototype.  More  often  improvements  were  made  between  experiments,  as  in 
the  expansion  of  new  features  on  the  C^  prototype  from  53  to  95  between  Experiments  2-4.  For 
more  responsive  feedback,  ARI  provided  a  Quick  Reaction  Report  to  PM  FCS  C^  approximately 
30  days  after  each  experiment  that  summarized  participants’  responses  on  subjective  measures  to 
document  and  expedite  the  improvements  requested. 

The  Interim  Reports  from  ARI  provided  to  PM  FCS  C^  approximately  90  days  after  each 
experiment  included  a  section  titled  “Recommendations  for  Future  FCS  C^  Experiments.”  This 
section  would  typically  identify  key  sustain  and  improve  recommendations  that  covered  a  range 
of  topics  including  experimental  design,  performance  measurement  and  the  distillation  and 
dissemination  of  findings. 

Notably,  many  of  these  recommendations  should  apply  to  other  research  efforts,  and 
particularly  research  on  future  battle  command  and  on  the  C^  systems  under  development  in  FCS 
simulation  and  acquisition  efforts.  Accordingly,  a  sample  set  of  recommendations  to  improve 
future  research  efforts  are: 

■  Invest  as  much  in  training  and  evaluation,  as  simulation. 

■  Employ  a  deliberate  practice  design  to  base  findings  on  proficient  performance. 

■  Use  simulation-based  training  exercises  to  develop  tactical  and  technical  skills. 
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■  Provide  structured  and  progressive  training  from  individual,  to  intra-unit,  to  cross-unit 
levels. 

■  Vary  mission  complexity  levels  to  determine  performance  limits. 

■  Assess  the  full  range  of  performance  including  planning  and  execution  phases. 

■  Include  novel  missions  and  terrain  to  broaden  the  spectrum  of  operations. 

■  Balance  subjective  measures  about  performance  with  objective  measures  of  performance. 

■  Capitalize  on  the  requirement  for,  and  value  of,  automated  measures  of  performance. 

■  Employ  methods  to  identify  and  codify  new  approaches  to  command  and  control  by 
documenting  the  lessons  learned  during  After  Action  Reviews  (e.g.,  TTPs  and  SOPs). 

■  Disseminate  findings  across  doctrine,  organizations,  training,  materiel,  leadership, 
personnel,  and  facilities  (DOTMLPF). 

■  Ensure  technology  complements  human  performance. 

Transfer  Measurement  Methods 

It  may  be  too  early  to  assess  the  suecess  of  our  effort  to  transfer  the  human  performance 
measurement  methods  developed  to  future  research  efforts.  However,  the  authors  conclude  that 
the  research  effort  was  successful  in  documenting  and  refining  the  measurement  methods  in  a 
manner  that  should  facilitate  transfer  to  a  broad  range  of  future  research  efforts  on  command  and 
control,  particularly  battle  command  and  command  group  research  issues. 

Subjective  Measures 

For  subjeetive  measures,  the  wide  range  of  questionnaires  and  interviews  including  the 
In-Place  AAR  that  were  developed,  adapted,  and  refined  by  ARI  are  fully  documented  (Lickteig, 
Sanders,  Durlach,  &  Carnahan,  in  preparation).  As  indicated,  these  measures  were  used  to 
obtain  participant  feedback  on  multiple  research  issues  including:  training,  skill  proficiency, 
workload,  performance  success,  teamwork  skills,  decision  making,  function  and  task  allocations, 
prototype  effectiveness,  and  human-system  integration. 

The  transition  of  most  subjective  measures  to  electronic  format  by  Experiment  4  should 
facilitate  the  transfer  of  measurement  methods  across  future  research  efforts.  Overall,  electronic 
administration  worked  fairly  well,  however,  two  shortcomings  were  noted  in  ARI’s  Quick 
Reaction  report  for  Experiment  4.  The  recommendations  provided  below  to  redress  these 
shortcomings  may  apply  to  future  research  on  battle  command  and  systems. 

First,  the  goal  of  in-place  administration  (in  the  Cf  vehicle)  was  not  met.  After  each  run, 
participants  were  escorted  to  a  remote  area  in  the  building  to  complete  questionnaires.  Second, 
and  more  importantly,  while  participants  were  completing  the  questionnaires  they  did  not  have 
access  to  a  prototype,  particularly  a  prototype  depicting  the  operational  run  just  completed. 
Future  efforts  should  ensure  participant  access  to  a  prototype  during  administration  to  review 
their  operational  situation  as  well  as  the  tools  and  features  they  are  being  asked  to  evaluate. 
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Objective  Measures 

For  objective  measures,  more  automated  approaches  to  identifying  and  coding  verbal  and 
particularly  human-computer  interaction  are  sorely  needed.  As  noted,  automated  measures  of 
performance  to  support  training  and  evaluation  are  a  formal  FCS  requirement  (Unit  of  Action 
Maneuver  Battle  Laboratory,  2003).  The  research  reinforced  that  requirement  by  quantifying  the 
HCI  analyst  workload  in  identifying  and  tabulating  over  1,500  interactions  by  the  command 
group  participants  in  an  experimental  run.  Particularly,  the  development  of  automated  measures 
will  greatly  facilitate  the  transfer  of  measurement  methods  to  future  training  and  evaluation 
efforts. 


Typically,  the  development  and  validation  of  automated  measures  is  an  iterative  process. 
This  process  generally  requires  the  collaborative  efforts  of  behavioral,  technical,  and  operational 
subject  matter  experts.  Behavioral  scientists  often  initiate  the  process  by  identifying,  describing, 
and  defining  the  measures  of  interest.  Technical  experts  then  attempt  to  develop  the  measures 
specified,  including  the  software  codes  required  to  identify  and  log  the  measures  as  defined.  As 
reported,  there  are  often  discrepancies  between  behavioral  scientist  inputs  and  technical  engineer 
outcomes.  Resolving  such  discrepancies  requires  additional  refinements,  often  by  the  behavioral 
and  technical  experts.  Ultimately,  operational  experts  must  help  determine  the  practical  utility  of 
the  automated  measures  developed  and  validated. 

Only  minimal  progress  was  made  toward  the  goal  of  developing  and  validating  automated 
measures  for  human-computer  interaction  during  Experiments  3  and  4.  The  approach  taken  and 
lessons  learned,  however,  were  documented  in  this  report  to  imderscore  the  work  required  and 
the  potential  payoff  of  automating  many  of  the  human-machine  performance  measures  integral  to 
future  command  and  control.  Clearly,  additional  development,  refinement,  and  validation  are 
needed  to  develop  a  useful  set  of  automated  measures  to  support  training,  evaluation,  and 
system  design.  In  particular,  automated  measures  are  needed  to  measure  the  process  of  learning 
to  provide  the  feedback  instrumental  to  learning  in  more  complex  eind  dynamic  settings. 

Automated  measures  of  verbal  interaction  were  not  attempted  for  Experiments  1-4. 
However,  the  results  from  verbal  analysis  (for  example  Figure  10)  stress  the  vital  role  of  verbal 
communication  in  battle  command  collaboration,  even  with  advanced  information  technologies. 
Recently,  the  development  of  automated  mathematical  analysis  tools,  including  Latent  Semantic 
Analysis  has  resulted  in  powerful  statistical  methods  for  extracting  and  representing  the  meaning 
of  words  and  phrases  (Landauer,  &  Dumais,  1997). 

The  authors  conclude  such  tools  may  provide  automated  methods  for  chunking  and 
coding  verbalizations  into  more  meaningful  classifications,  including  the  function  used  to 
categorize  participant  verbalizations  for  FCS  C^.  Importantly,  ARI’s  documentation  on  methods 
for  chunking  and  coding  verbalizations  should  greatly  facilitate  development  of  such  tools  for 
command  and  control  research  efforts. 
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Develop  Innovative  Training 


A  final  objective  was  to  identify  training  requirements  and  develop  innovative  training 
approaches  based  on  empirical  measures  of  command  group  performance.  Success  on  this 
objective  is  a  primary  focus  of  the  ongoing  work  by  ARI  to  address  the  STO  Objective  “Methods 
and  Measures  of  Commander-Centiic  Training.”  As  a  complement  to  the  PCS  research,  ARI 
is  developing  an  in-house  laboratory  to  help  develop  the  innovative  training  required  by  PCS, 
including  simulation-based  training  and  structured,  embedded  training  (Lickteig,  et  al.,  2002). 
Two  aspects  of  ARI’s  in-house  training  development  efforts  most  directly  related  to  the  PCS 
program  will  be  briefly  described. 

Training  Requirements  and  Approaches 

Training  requirements  and  recommendations  for  future  command  groups  were  a  primary 
focus  of  the  previously  noted  ARI  research  comparing  novice  versus  expert  participants  in  the 
PCS  C^  research  environment  (Carnahan,  Lickteig,  Sanders  &  Durlach,  in  preparation).  As 
expected,  significant  differences  in  tactical  and  technical  performance  between  novices  versus 
experts  were  obtained.  Based  on  the  findings,  and  literature  on  novices  versus  experts,  a  set  of 
training  recommendations  were  presented  to  help  turn  novices  into  expert  command  groups. 

This  set  of  training  recommendations  is  re-presented  below  in  Table  6  to  provide  a  more 
complete  account  of  ARI’s  PCS  C^  research  efforts  in  this  report.  Although  briefly  described 
here,  more  complete  description  and  discussion  of  these  recommendations  is  provided  in 
Carnahan,  Lickteig,  Sanders  and  Durlach  (in  preparation). 

Overall,  the  recommendations  in  Table  6  reflect  ARI’s  basic  conclusion  that  the  training 
requirements  for  PCS  command  groups  entail  unprecedented  levels  of  tactical  and  technical 
expertise.  These  recommendations  stress  the  need  to:  master  basic  skills,  train  part-task,  ensure 
consistency,  and  provide  feedback.  They  reinforce  new  PCS  training  requirements,  including 
structured  and  embedded  training,  for  transforming  novices  into  expert  command  groups. 

Por  each  recommendation,  training  implications  and  examples  from  PCS  C^  research  are 
identified.  These  examples  were  based  on  training  problems  observed  in  the  PCS  C^  experiments 
across  novice  and  expert  participants.  The  authors’  intent  was  to  provide  training  development 
efforts  more  specific  and  tangible  instances  of  future  training  problems  and  requirements. 

A  Prototype  Training  Example:  Battle  Command  Visualization  101 

As  an  example  of  ongoing  training  development  efforts  by  ARI,  the  training  example 
called  Battle  Command  Visualization  (BCV)  101  in  Table  6  will  be  briefly  summarized  here. 

With  support  from  the  PCS  C^  program,  a  prototype  training  program  called  BCV  101  is  being 
developed  by  ARI  to  provide  simulation-based  and  embedded  training  on  basic  visualization 
skills  required  for  future  battle  command. 

Recall  that  the  See  function  required  the  command  group  participants  to  control  and 
interpret  input  from  an  array  of  marmed  and  unmanned  networked  sensors  to  maintain  an 
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accurate  battlefield  “picture.”  However,  empirical  measures  of  command  group  performance 
disclosed  that  novice  and  even  expert  participants  had  repeated  difficulty  in  understanding  and 
exploiting  the  capabilities  and  limitations  of  the  sensor  systems  available  to  the  Unit  Cell. 

For  the  current  force,  serious  shortcomings  in  visualization  training  and  skill  are  well 
documented,  particularly  at  command  and  staff  levels  (Reilley,  1997;  Solick,  1997).  In  general, 
training  on  visualization  is  not  readily  available  or  provided,  and  lecture-based  training  on 
visualization  is  not  efficient  or  effective. 

Table  6 


Training  Recommendations,  Implications  and  Examples  for  FCS  Command  Groups 


Training  Recommendations 

Training  Implications 

Training  Examples 

Master 

Basic  Skills 

Tactical  Skills 

Technical  Skills 

Integrate  Tactical  and  Technical 

METT-TC 

HCI  Taxonomy 

Intervisibility  Plotting 

Train 

Segmentation 

LAM  Fire  Execution 

Part-Task 

Simplification 

Reduce  Duty  Position  Requirements 

Ensure 

Consistent  Conditions 

Battle  Command  Visualization  101 

Consistency 

Consistent  Venues 

System/Simulation/Personal 

Computer  (PC)/Personal  Data  Assistant 
(PDA) 

Timeliness 

System  Alerts  and  Mentor 

Provide 

Standardization 

Systematic  and  Comprehensive 

Feedback 

Diagnostic  Precision 

“Personalize”  Feedback 

Presentation 

Tailor  Presentation  Mode 

For  the  future  force,  the  tactical  and  technical  skills  required  to  See  First  are  formidable 
as  indicated  by  results  from  the  FCS  &  program.  The  FCS  goal  that  small  eommand  groups  will 
eontrol  a  wide  range  of  manned  and  unmanned  networked  sensors  and  correetly  interpret  the 
outputs  from  these  sensors  substantially  raises  skill  requirements.  For  example,  a  potential  set  of 
sensor  outputs  that  must  be  correctly  interpreted  includes:  direct  view  and  infrared  images  from 
unmanned  aerial  and  ground  vehicles,  moving  target  indicators,  synthetic  aperture  radar  pictures, 
eounter-fire  target  acquisition  and  ground  surveillance  radar  detections,  and  signature  results 
from  acoustic,  communication,  and  electronic  sensors. 


New  training  approaches  and  programs  are  needed  to  provide  the  tactieal  and  technical 
skills  required  by  FCS.  The  BCV  101  focuses  on  the  skills  required  to  control  and  interpret 
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sensor  systems,  identified  here  as  basic  battle  command  visualization  skills  for  PCS  command 
groups. 


Skill  mastery,  or  reliably  high  levels  of  performance,  is  generally  the  product  of 
consistent  training  conditions  and  feedback.  Research  on  automaticity  stresses  that  consistent 
conditions  and  feedback  are  needed  to  provide  experts  the  ability  to  perform  basic  skills  through 
automated  processing  (Fisk  &  Rogers,  1992).  Notably,  computer-based  and/or  embedded 
training  coupled  with  simulation  are  ideal  mediums  for  ensuring  consistent  training  conditions 
and  feedback. 

The  BCV  101  training  will  provide  consistent  task  conditions  and  feedback  across 
different  training  venues,  or  mediums,  any  time  and  anywhere.  The  performance-oriented  nature 
of  BCV  101  training  will  exploit  the  power  of  digital  systems,  including  embedded  systems  to 
perceptually  augment  sensor  coverage  and  simulation  to  provide  realistic  performance  feedback 
on  sensor  outputs. 

The  BCV  101  prototype  training  development  effort  will  address  two  primary  objectives 
for  providing  basic  battle  command  visualization  skills  for  PCS  command  groups: 

■  Develop  a  representative  subset  of  training  exercises  to  control  and  interpret  individual 

sensors  and  complementary  networked  sensors. 

■  Design  an  overall  training  program  to  provide  a  comprehensive  and  progressive  set  of 

training  exercises  modeled  on  the  Conduct  of  Fire  Trainer  (COPT)  training  program. 

The  COPT  program  for  tank  gunnery  uses  simulation  to  help  commanders  and  gunners 
develop  and  practice  their  firing  skills  in  a  variety  of  increasingly  difficult,  day  and  night 
scenarios  (U.S.  Army  Armor  School,  2003).  Progressively  structured  exercises  range  fi-om 
single,  daylight,  stationary,  near-range  targets  to  multiple,  night-time,  moving,  far  range  targets; 
fi-om  working  as  an  individual  vehicle  to  fighting  as  part  of  a  larger  unit.  Training  progression 
and  skill  proficiency  are  systematically  controlled  and  assessed  through  a  series  of  pass  the  gate 
exercises. 

Design  for  the  BCV  101  training  program  is  committed  to  providing  a  consistent  multi- 
media  approach  to  training  battle  command  visualization  skills.  The  design  will  be  layered 
across  training  venues  to  include:  wireless  personal  data  assistants,  desk-  and  lap-top  computers, 
virtual  simulation  facilities,  internet-linked  gaming  sites,  and  fully  embedded  training  in 
operational  systems.  Layers  across  media  may  correspond  to  crawl,  walk,  and  run  training  levels 
for  individual  and  collective  battle  command  visualization  skills.  A  goal  of  the  BCV  101 
prototype  training  development  effort  is  to  provide  a  cornerstone  example  of  the  simulation- 
based,  embedded  training  required  for  PCS. 

Conclusions 

This  section  provides  some  preliminary  conclusions  about  human  performance  issues  for 
future  battle  command  based  on  the  PCS  C^  research  program.  More  problematic  conclusions  on 
workload  and  training  are  followed  by  more  promising  conclusions  on  user-based  involvement 
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and  proactive  research,  particularly  their  potential  for  solving  many  of  the  problems  anticipated 
in  workload  and  training. 


Workload 

The  human  performance  findings  from  Experiments  1-4  suggest  workload  may  be  a 
serious  problem  for  future  command  groups  in  small  PCS  units.  The  negative  impact  of 
workload  on  performance  was  underscored  by  participants’  low  ratings  of  their  own  performance 
success  after  more  complex  runs.  Objective  data  confirmed  that  participants  were  heavily 
engaged  in  verbal  and  human-computer  interactions  during  more  complex  runs  and  more  intense 
run  segments. 

Automation  may  eventually  reduce  workload  but  it  is  a  double-edged  sword  in  early 
development  of  a  system  such  as  PCS.  A  striking  example  of  how  automation  can  reduce 
workload  was  a  comparison  on  number  of  same  sensor  images  viewed  at  different  times  and  by 
dilTercnt  participants  in  Experiment  2  versus  Experiment  3.  Por  Experiment  3,  an  automated 
audit  trail  on  images  viewed  was  introduced  that  clearly  indicated  if,  when,  and  by  whom  each 
image  had  been  opened  and  viewed.  This  automated  feature  reduced  the  number  of  same  images 
viewed  in  Experiments  2  versus  3  from  62%  to  16%. 

Notably,  the  overall  decrease  in  workload  and  interaction  between  Experiments  2  and  3 
was  attributed  to  the  array  of  new  and  increasingly  automated  features  added  to  the  &  prototype 
for  Experiment  3  based  on  results  such  as  the  reduction  in  same  images  reviewed. 


Why  then  the  increase  in  workload  and  interaction  in  Experiment  4  with  more  automated 
features  and  more  experienced  participants?  One  plausible  explanation  is  that  programmatic 
expectations  about  baseline  capabilities  including  faster  mission  completion  may  have  raised 
perfi>rmance  requirements.  The  “High”  complexity  run  requirements  of  Experiment  4  may  have 
actually  exceeded  the  “High”  and  even  “Too  High”  levels  of  Experiments  1-3  because  the 
participants  were  repeatedly  urged  to  reach  the  objective  for  each  run  faster. 

If  so,  the  increase  in  participant  workload  during  Experiment  4  may  be  an  artifact 
common  to  exploratory  versus  controlled  research,  and  a  reality  common  to  expectations  about 
doing  “more  with  less”  through  technology  (Cordesman  &  Wagner,  1996). 

Training 

Training  requirements  for  PCS  command  groups  may  entail  unprecedented  levels  of 
tactical  and  technical  expertise.  Given  the  PCS  organization  examined,  a  small  command  group 
with  highly  automated  and  interdependent  network-based  systems  must  reformulate  battle 
commands  into  computer  commands.  More  succinct  verbalizations — such  as  commander’s 
intent  and  guidance— with  many  implied  tasks,  must  be  issued  in  computer-mediated/dictated 
formats  in  which  most  tasks  must  be  explicitly  and  precisely  defined. 

More  automated  features  such  as  Auto  Recon  and  Route  Generation  for  unmanned 
platforms  may  relieve  players  of  routine  tasks  such  as  entering  route  points  or  doing  repeated 
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terrain  analysis  checks  on  intervisiblity  and  mobility.  However,  automation  often  raises  the 
demand  on  players  to  understand  the  decision  rules  and  parameters  designed  into  automated  task 
features  Jind  to  monitor  task  execution.  For  example,  during  Experiment  3  the  Shadow  UAV 
tended  to  “wander  off’  toward  enemy  elements  under  Auto  Recon  and  be  destroyed,  a  critical 
loss  to  the  See  First  capability  of  the  Unit  Cell.  This  unintended  consequence  of  high  automation 
was  due  to  misunderstood  recon  parameters,  information  overload  that  limited  monitoring,  and 
lack  of  an  effective  human  override  to  abort  an  automated  routine  in  a  timely  manner. 

Understanding  the  limits  and  strengths  of  technology  was  a  severe  challenge  to  relatively 
expert  participants.  A  key  part  of  the  challenge  was  to  comprehend  the  user  input  requirements 
and  operational  consequences  for  more  automated  functions  including  Plan,  See,  Move,  and 
Strike.  After  Experiment  4,  participants  who  had  completed  40+  experimental  runs  across  the 
four  experiments  expressed  strong  concerns  about  training  and  their  ability  to  fully  exploit  Unit 
Cell  capabilities.  Moreover,  the  high  rank  and  operational  experience  of  the  participant 
command  group  will  be  the  exception,  not  the  rule,  in  small  FCS  units. 

The  ARI’s  conclusions  on  training  stress  the  need  for  three  basic  but  far-reaching 
improvements  in  the  training  of  individuals  and  units  in  future  FCS  organizations,  and 
particularly  command  groups; 

■  Develop  progressive  simulationrbased  training  exercises  directed  at  individual  tactical 
and  technical  skills,  particularly  user  input  requirements  for,  and  the  operational 
consequences  of,  automated  functions  including  Plan,  See,  Move,  and  Strike. 

■  Develop  a  parallel  set  of  collective  command  group  training  exercises  directed  at  intra- 
unit  tactical  and  technical  skills. 

■  Develop  a  parallel  set  of  multi-echelon,  distributed  training  exercises  directed  at  cross- 
unit  tactical  and  technical  skills. 

User-Based  Involvement 

The  interface  is  increasingly  the  primary  locus,  or  means,  of  interaction  between  war 
fighters  and  systems.  Developing  an  optimal  interface  requires  intense  user-based  involvement, 
not  a  bunch  of  guys/gals  sitting  aroimd  a  table  (BOGS  AT)  or  a  drive-by  user  jury. 

A  distinctive  hallmark  of  the  FCS  research  program  is  the  sustained  use  and 
refinement  of  a  prototype  system  and  interface  by  an  exceptional  set  of  command  group 
participants.  For  over  two  years,  the  command  group  participants  led  the  Operational  Team’s 
committed  and  sustained  effort  to  design,  test,  and  develop  the  unique  and  challenging  system 
and  interface  required  for  FCS. 

As  a  result,  the  FCS  C^  program  continues  to  explore  and  develop  a  cutting-edge  C^ 
system  and  interface  for  the  command  and  control  challenges  raised  by  the  emerging  concepts  of 
FCS  (Unit  of  Action  Maneuver  Battle  Laboratory,  2003).  Much  work  remains,  but  unlike  many 
other  fielded  and  prototype  C^  systems,  the  FCS  C^  system  and  interface  provide: 


41 


■  Value  grounded  by  experts’  sustained  use  and  refinement. 

■  An  increasingly  effective  interface  to  autonomous  systems. 

■  A  common  interface  for  human  and  robotic  forces. 

Proactive  Research 

Historically,  Army  acquisition  research  has  had  difficulty  conducting  an  adequate  early 
assessment  of  the  human  dimension  in  system  performance.  This  human  performance  issue  is 
especially  critical  for  PCS  because  the  empowerment  of  the  commander  through  advanced 
systems  is  at  the  heart  of  the  PCS  concept.  Moreover,  the  revolutionary  nature  of  the  Army’s 
transfomation  embodied  in  the  PCS  acquisition  program  increases  the  risk  of  relying  exclusively 
on  traditional  assessment  methods  such  as  C  hardware  and  software  component  tests,  or  the 
outcomes  of  simulation  without  Soldiers-in-the-loop. 

The  PCS  C^  program  exemplifies  the  proactive  research  on  human  performance  that  is 
essential  to  improving  human  system  integration.  The  ARI’s  methods  and  results  on  human 
performance  provided  reliable  and  empirical  data  for  important  and  timely  decisions  on  training, 
materiel,  manpower,  and  personnel.  Pindings  were  also  readily  transitioned  to  acquisition  efforts 
because  of  DARPA’s  dual  roles  in  PCS  simulation  and  acquisition.  The  PCS  C^  program  was 
cited  by  the  PCS  Lead  System  Integration  team,  the  PCS  Integrated  Product  Team  for  Training, 
as  a  key  contributor  to  their  design  plarming.  Pindings  shaped  the  C^  prototype  showcased  in  the 
PCS  Capstone  Demonstration  of  C^  systems  prior  to  the  PCS  Milestone  B  decision. 

The  ultimate  value  of  a  research  and  development  program  is  determined  as  much  by  the 
resources  spent  on  training  and  evaluation,  as  the  resources  spent  on  simulation.  The  ultimate 
value  of  PCS  and  future  C^  systems  is  determined  not  so  much  by  technology,  but  by  shaping 
technology  to  complement  human  performance. 
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Appendix  A 
List  of  Acronyms 


AAR 

AFRU 

AGM 

AH-64D 

ARI 


BCV 

BDA 

BOGSAT 

C^ISR 

CECOM 

COA 

COFT 

CSE 

DARPA 

DCSOPS&T 

DIS 

DOTMLPF 


FBC 

FCS 

FCS 

GCM 

HCI 

HTR 

HUD 

IFV 

LAM 

LOS 


After  Action  Review 
Armored  Forces  Research  Unit 
Attack  Guidance  Matrix 
Attack  Helicopter-64D 

U.S.  Army  Research  Institute  for  the  Behavioral  and  Social 
Sciences 

Battle  Command  Visualization 

Battle  Damage  Assessment 

Bunch  of  Guys/Gals  Sitting  Around  a  Table 

Command  and  Control 

Command,  Control,  Communications,  Computers,  Intelligence, 
Surveillance,  and  Reconnaissance 
Communications  Electronics  and  Engineering  Command 
Course  of  Action 
Conduct  of  Fire  Trainer 
Commander’s  Support  Environment 

Defense  Advanced  Research  Projects  Agency 
Deputy  Chief  of  Staff  for  Operations  and  Training 
Distributed  Interactive  Simulation 

Doctrine,  Organizations,  Training,  Materiel,  Leadership,  Personnel,  and 
Facilities 

Future  Battlefield  Conditions 

Future  Combat  Systems 

Future  Combat  Systems  Command  and  Control 

Graphic  Control  Measure 

Human-Computer  Interaction 
human  target  recognition 
Heads-Up  Display 

Infantry  Fighting  Vehicle 

Loitering  Attack  Missile 
Line  of  Sight 


A-1 


METT-TC 

Mission,  Enemy,  Terrain,  Troops,  Time  and  Civilians 

NAI 

Named  Area  of  Interest 

NASA 

National  Aeronautics  and  Space  Administration 

NTC 

National  Training  Center 

0/C 

Observer/Controller 

OneSAF  OTB 

One  Semi-Automated  Forces  Testbed  Baseline 

PAM 

Precision  Attack  Missile 

PC 

Personal  Computer 

PDA 

Personal  Data  Assistant 

PM 

Program  Manager 

RDEC 

Research,  Development  and  Engineering  Center 

SAMS 

School  of  Advanced  Military  Studies 

SME 

Subject  Matter  Expert 

SOP 

Standing  Operating  Procedure 

STO 

Science  and  Technology  Objective 

TLX 

Task  Load  Index 

TRADOC 

Training  and  Doctrine  Command 

TIP 

Tactics,  Techniques  and  Procedures 

UAV 

Unmanned  Aerial  Vehicle 

UGS 

Unmanned  Ground  Sensors 

WIMP 

Windows,  Icons,  Menus,  and  Pull  Downs 

A-2 


